test-infra: Unable to fetch/merge certain pull requests

What happened: Some presubmits are failing because they either:

can’t fetch a given PR’s ref

$ git fetch https://github.com/kubernetes/kubernetes.git pull/96426/head
fatal: couldn't find remote ref pull/96426/head

can fetch a ref, but it doesn’t match what GitHub’s API is saying

$ cat .git/FETCH_HEAD
1a439c03e242a0735f53bf54e0ac2c999844d836		'refs/pull/95921/head' of https://github.com/kubernetes/kubernetes
$ hub api /repos/kubernetes/kubernetes/pulls/95921 | jq -r .head.sha
92383b25ba1455d2ddc866a57b1b7e249c9b04f8

These errors are reproducible locally

What you expected to happen:

To be able to fetch a pull request from github, and have its HEAD SHA agree with github’s API

How to reproduce it (as minimally and precisely as possible):

$ git init
$ git fetch https://github.com/kubernetes/kubernetes.git pull/96426/head
fatal: couldn't find remote ref pull/96426/head

$ git init
$ git fetch https://github.com/kubernetes/kubernetes.git pull/95921/head
$ git show $(hub api /repos/kubernetes/kubernetes/pulls/95921 | jq -r .head.sha)
fatal: bad object 92383b25ba1455d2ddc866a57b1b7e249c9b04f8

Please provide links to example occurrences, if any:

Anything else we need to know?:

I strongly suspect this is a problem on GitHub’s side. Mostly making this issue to reference in areas where this is happening.

One possible workaround / resiliency fix would be to ignore GitHub’s API when it comes to choosing which SHA to checkout for which pull request.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 22 (22 by maintainers)

Commits related to this issue

Clonerefs: Fetch sha instead of pull head if available Mitigates https://github.com/kubernetes/test-infra/issues/19910 and avoid races where the pull head differs from the SHA because a push occured ... — committed to alvaroaleman/test-infra by alvaroaleman 4 years ago

Most upvoted comments

Just to update from Github’s side - they are still investigating. They are aware of this thread and are following it.

mrbobbytables on Nov 12, 2020

Filed a support ticket with GitHub and cc’ed kubernetes github admin team, as I’m going to be unavailable the rest of the week

spiffxp on Nov 10, 2020

One possible workaround / resiliency fix would be to ignore GitHub’s API when it comes to choosing which SHA to checkout for which pull request.

That’s not an option. The tests (AKA prowjob custom resources) contain the exact SHAs that get tested and we need that information to be accurate in order to determine what revisions got tested.

alvaroaleman on Nov 10, 2020

@mrbobbytables if it’s at all possible I think it might be useful to gently push back on the idea that this is “not recommended” - it’s an API they provide explicitly for this purpose and their official tooling uses it. Hopefully such a “not recommended” stance is not GitHub Engineering saying that this won’t be fixed…

stevekuznetsov on Nov 12, 2020