agent: Occasional checkout fails

We’re experiencing a weird git checkout issue. This error has been increasing in frequency recently. Any idea what is going on here?

Preparing build folder  0.2s
$ mkdir -p "/var/lib/buildkite-agent/builds/{{git_checkout_path}}"
$ cd "/var/lib/buildkite-agent/builds/{{git_checkout_path}}"
$ git remote set-url origin "git@github.com:{{repo}}.git"
$ git clean -fdq
$ git submodule foreach --recursive git clean -fdq
$ git fetch origin "+refs/pull/1705/head:"
From github.com:{{repo}}
 * branch            refs/pull/1705/head -> FETCH_HEAD
$ git checkout -qf "d0de339b3879631a6d6cd661fa900749ab5201de"
fatal: reference is not a tree: d0de339b3879631a6d6cd661fa900749ab5201de

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 20 (11 by maintainers)

Most upvoted comments

@cactus the agent is working as expected, but it’s seems there an issue with GitHub at the moment.

It seems the refs/pull/{id}/head ref fails to update if you have branch merge protection turned on for your GH repo, and “dismiss stale pull requests” enabled. The ref can get stuck to the commit before the first review dismissal.

The workaround is to force push new commits to the PRs branch, or turn merge protection off.

With the help of someone who managed to reproduce it, we’ve recreated the bug here: https://github.com/buildkite/pull-request-ref-head-bug/pull/1

git clone https://github.com/buildkite/pull-request-ref-head-bug.git
cd pull-request-ref-head-bug
git fetch origin refs/pull/1/head
git rev-parse FETCH_HEAD

You can see the rev-parse returns a6f10f96 instead of 6cd1079d.

GitHub will keep us posted on a fix, and I’ll keep this ticket updated as well.

toolmantim on Apr 7, 2018

no not really, the support was able to help by resetting some cache that caused some troubles in this case. We apparently ran into an edge case bug of the backend. 🙈 It’s all solved now and running smooth for a few days.

taschik on Nov 20, 2019

Follow-up from GitHub:

Just a quick update to let you know that we’ve deployed a fix for this issue. Would you mind testing this out on your end? Please do let us know if you are still seeing this problem.

pda on Apr 19, 2018