cli: gh repo sync not working

Hello everyone, thank you for reporting this bug and providing additional information to help out.

I believe this is a platform change, but I also believe you can resolve it through the CLI. If you aren’t interested in the details, jump to the Resolution section below.

Investigation

Firstly, the 404 Not Found on https://api.github.com/repos/<org>/<repo>/git/refs/heads/<branch> is a bit of a red herring. This 404 is a failure to PATCH, while attempting to update the branch to point at the commit ref provided in the body. I believe this is failing because the commit does not exist on the targeted repository. The reason it doesn’t exist on the provided repository is due to an earlier failure on the /merge-upstream endpoint (that is swallowed for historical reasons I don’t yet understand). For example, from @andrewpollock’s logs:

> POST /repos/andrewpollock/osv.dev/merge-upstream HTTP/1.1
> Host: api.github.com
...

{
  "branch": "master"
}

< HTTP/2.0 422 Unprocessable Entity

Edit: My assumption around the 404 was incorrect, see the following post for details about why it failed and why the 422 and 404 have the same cause.

Why is this endpoint failing (intermittently)?

In digging into this, I found a recent change that began requiring workflow scope on the token if any commits to be merged from upstream included workflow changes.

I was able to somewhat prove this locally by forking ocaml/dune, checking out @Alizter’s commit, and seeing gh repo sync williammartin-test-org/dune fail because the next commit included a workflow change. I then reset to and pushed HEAD^1 from main and synced successfully, seeing that the next commit didn’t have a workflow change.

I believe this theory is further evidenced by:

• Reports that this occurs intermittently (it would only occur if upstream commits to be merged included a workflow change)
• Reports that using sync repo button on the UI succeeds (which would already have the required privileges)
• Reports that after using sync repo on the UI, that the CLI works again (because the merge-upstream no longer fails and the commit exists after PATCH occurs)
• Adding the workflow scope to my token allowed me to sync even with workflow changes upstream.

Resolution

To resolve this you can add the workflow scope to your token either on initial login like so:

gh auth login -s workflow

or via refreshing you existing token like so:

gh auth refresh -s workflow

Caveat

It’s possible that this change introduced other failure cases that I don’t yet have the necessary knowledge to understand but I believe that this is almost certainly the common case. If after refreshing your token you still experience errors, please leave a comment.

CLI Changes

From our point of view, we should determine how to handle this error in a more graceful manner, prompting to refresh to include the token. Unfortunately, there are a number of reasons that the API returns 422 and the reasons are opaque to clients, so we may need some changes before we can support this.

In the meantime, hopefully this issue serves as a place for people to find help.

@spencerschrock Sounds like a platform bug that permissions: write-all does not allow syncing. I can ask around internally about it.

Update, I heard back from the internal API team and they explained that this behavior is a security feature and is not a bug. So unfortunately the only work around is to generate a new token, perhaps with this, that has the correct scopes.

Further Investigation

Following up on the previous comment I did some further digging into the relationship between the earlier 422 and the later 404. As it turns out, both occur for the same reason as far as I can tell (missing workflow scope).

I created a commit that changed a workflow that was not associated with a branch and tried to PATCH update the head/main ref and received a 404 when using a token without workflow scope:

It was later successful with workflow scope:

What’s happening here is that as described in the oauth scopes doc, the workflow scope is needed for any workflow file change that isn’t in a branch on the repository already. When we attempt to update the ref, GitHub is rejecting this as “you don’t have permission to point this branch at something that contains a workflow file change” and that’s manifesting as a 404.

I think that ties up the loose ends!

After I’ve synced by the web UI:

$ DEBUG=1 gh repo sync andrewpollock/osv.dev
* Request at 2023-06-14 09:05:27.150894162 +1000 AEST m=+0.065482152
* Request to https://api.github.com/graphql
⣷* Request took 919.547876ms
* Request at 2023-06-14 09:05:28.071390589 +1000 AEST m=+0.985978644
* Request to https://api.github.com/repos/andrewpollock/osv.dev/merge-upstream
⣻* Request took 296.000559ms
✓ Synced the "andrewpollock:master" branch from "google:master"

In case this helps:

$ DEBUG=1 gh repo sync andrewpollock/osv.dev
⣾* Request at 2023-06-14 09:03:50.551435814 +1000 AEST m=+0.081041738
* Request to https://api.github.com/graphql
⣷* Request took 916.596168ms
* Request at 2023-06-14 09:03:51.469220503 +1000 AEST m=+0.998826430
* Request to https://api.github.com/repos/andrewpollock/osv.dev/merge-upstream
⣻* Request took 379.934845ms
* Request at 2023-06-14 09:03:51.849800312 +1000 AEST m=+1.379406238
* Request to https://api.github.com/graphql
⣟* Request took 312.245235ms
* Request at 2023-06-14 09:03:52.162329298 +1000 AEST m=+1.691935223
* Request to https://api.github.com/repos/google/osv.dev/git/refs/heads/master
⣷* Request took 251.153768ms
* Request at 2023-06-14 09:03:52.41393556 +1000 AEST m=+1.943541558
* Request to https://api.github.com/repos/andrewpollock/osv.dev/git/refs/heads/master
⣽* Request took 304.482351ms
HTTP 404: Not Found (https://api.github.com/repos/andrewpollock/osv.dev/git/refs/heads/master)

Consider reopening - I have been encountering this off-and-on for at least a week.

Today it appears to work. I would guess then somebody in the GitHub API was tweaking some things causing my queries to fail. Nothing actionable from you guys I suppose.