kubernetes: git checkout fails with "reference is not a tree"

I see the following in Jenkins verification of https://github.com/kubernetes/kubernetes/pull/27453:

hudson.plugins.git.GitException: Command "/usr/bin/git checkout -f 6676629a96bf8631fbe106ff5ad147527359cbcc" returned status code 128:
stdout: 
stderr: fatal: reference is not a tree: 6676629a96bf8631fbe106ff5ad147527359cbcc

Artifacts at https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/27453/kubernetes-pull-verify-all/140

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 24 (24 by maintainers)

Most upvoted comments

edit: What I describe below is true and can happen, but most of our cases are caused by a race condition in the Jenkins plugin and aren’t caused by anyone force-pushing.

Oh! I think I remember this one. I think what happens is this:

You open a PR whose head has sha abcde123.
Jenkins queues up a build with that sha, but doesn’t run it yet.
You force-push over that commit, so the head of your PR is now something else.
Jenkins tries to checkout the old sha, which no longer exists.

It might not be exactly that, since we actually try to check out the merge commit, but I think that’s the idea.

spxtr on Jul 12, 2016

It’s the Git plugin. I’ve patched it and installed the fixed version on pr-jenkins last night, and haven’t observed any failures since then. Normally there are ~20 failures per day.

See JENKINS-26290 for the full details.

rmmh on Sep 9, 2016

Latest version of the plugin (1.32.8) is broken in a new and fun way:

$ while true; do curl -slL https://api.github.com/rate_limit?access_token=[elided] | jq .resources.core.remaining ; sleep 1; done
592
591
589
587
586
584
582
575
564
553
545
535
529
519
509
500
491
483
478
476
473

Looking at the logs, it seems to be repeatedly querying PRs without abandon; I see lines like

Jul 05, 2016 10:05:04 PM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #16,062 was updated on repo kubernetes/kubernetes but there aren't any new comments nor commits; that may mean that commit status was updated.
Jul 05, 2016 10:05:04 PM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #13,925 was updated on repo kubernetes/kubernetes but there aren't any new comments nor commits; that may mean that commit status was updated.
Jul 05, 2016 10:05:04 PM INFO org.jenkinsci.plugins.ghprb.GhprbPullRequest updatePR
Pull request #13,216 was updated on repo kubernetes/kubernetes but there aren't any new comments nor commits; that may mean that commit status was updated.

repeating a few times per PR per minute.

Downgrading back to 1.31.4 again…

ixdy on Jul 6, 2016

So it looks pretty clear that the GitHub PR builder plugin doesn’t handle multiple executors very well. We try to check out origin/pr/${ghprbPullId}/merge, but increasingly frequently this is set to a different PR that we’re testing at the same time.

There was a rework of some of the envvar code recently, and I tried upgrading the plugin to see if that fixed the issue, but I then ran into a different as-of-yet undiagnosed issue where the plugin was failing to schedule any jobs at all.

I’m noticing that https://github.com/jenkinsci/ghprb-plugin/commit/9c46eab31437d8feaeec7658fcef5062f6b7a40c merged since then, and could very likely explain the more recent issue. So I might try upgrading the plugin again.

ixdy on Jul 5, 2016