Save commit data

[mihneadb]

Use http://developer.github.com/v3/repos/commits/#list-commits-on-a-repository together with the "since" parameter to perform optimized queries.

Should use the 'CommitData' type.

[adaschevici]

Can you explain how they should be optimized?
Do you mean only return data for the query for a certain period?

[mihneadb]

So.. right now, for issues, milestones and pull requests we query github for "all the entries" - since all of these can change state (they can get closed, reopened, updated etc.).
However, commits don't change - and they are also a lot (for reasonably sized projects). This means that if we requested all the commits for a project all the time, we would unnecessarily burn through our rate limit.

What I suggest is that on every update we ask github for the next 5000 commits from where we left off at the last update (the API endpoint supports pagination but it will only give you at most 100 commits per page, so that means 50 requests).

It seems that the API gives you commits in descending chronological order, but we want to start with oldest first (so we can have all of them and further updates only have to add the newest commits).

[mihneadb]

Also, thinking more about this, we could just get the most recent 5000 commits every time we do an update and drop the old ones. I'm not sure really old commits provide us any value. What do you think?

[adaschevici]

👍 on newer commits

[mihneadb]

OK, then let's do that. :)
It shouldn't be hard to implement. getData should have a max_pages attribute (which will be 50 for this usecase), and then probably you can just use it with the proper URL (api.../owner/repo/commits?par_page=100) to fetch the data.

Do you want to implement this?

[adaschevici]

Sure i'll have a hack at it.

[mihneadb]

Great! Let me know if I can help.

I m not sure that the regex for the header processing holds for the
next_page url for the commits endpoint. You'll find out :P

On Tue, Mar 4, 2014 at 12:19 PM, Artur Daschevici
notifications@github.comwrote:

Sure i'll have a hack at it.

Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-36614587
.

Mihnea Dobrescu-Balaur

[adaschevici]

The commits endpoint returns the results based on sha order and branch
top=branch_name&last_sha=sha_value
Not sure if they maintain the per_page pagination parameter across the requests because the subsequent requests come in without it.
And yes the parse headers breaks because of the different format.
I am thinking of adding in a type parameter to the getData, Maybe we should join the events and the commit data inside the same method.

[mihneadb]

The pagination aspect can be checked :). We can see how many we get and decide on a limit for that.

Why do you want to join the events and the commits?

[adaschevici]

They look somewhat similar, aside from the url steps are basically the same.

On Thu, Mar 6, 2014 at 12:08 PM, Mihnea Dobrescu-Balaur <
notifications@github.com> wrote:

The pagination aspect can be checked :). We can see how many we get and
decide on a limit for that.

Why do you want to join the events and the commits?

Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-36841134
.

Regards,
Arthur

[mihneadb]

Wait a sec - are we talking about the same thing?

getData has a type parameter already and it's used for all the endpoints:
https://github.com/uberVU/elasticsearch-river-github/blob/master/src/main/java/com/ubervu/river/github/GitHubRiver.java#L248

[adaschevici]

Ok.
My understanding was:
The function in the https://github.com/uberVU/elasticsearch-river-github/blob/master/data_processor/github.py#L87 where we get data for events.
What we want to do is add a method for getting commits at a limited rate in this function.

[mihneadb]

Well.. this is an issue in the elasticsearch-river-github repo, and you are talking about the elasticboard repo.
This is the elasticsearch plugin that fetches the github data into elasticsearch. The elasticboard/data_processor module doesn't fetch the data, it just queries it.

The github file you are referring to is there just with helpers for getting some data for prototype testing. I'll remove it since it's not used anymore.