build-push-action: Random 403 errors when pushing to GHCR

Troubleshooting

Pushing to GHCR seems to randomly resulting in 403s in the last 24 hours. Rerunning the GH action seems to resolve things (partially).

Behaviour

See above.

Failure message : buildx failed with: error: failed to solve: unexpected status: 403 Forbidden

Steps to reproduce this issue

  1. Kick off GH action to rebuild Docker images
  2. See https://github.com/jauderho/dockerfiles/actions
  3. Check one of the recent failed runs. For example, https://github.com/jauderho/dockerfiles/actions/runs/1255324536

Expected behaviour

Builds should complete with no issue.

Actual behaviour

See https://github.com/jauderho/dockerfiles/actions/runs/1255324536

Configuration

https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/tailscale.yml

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 7
  • Comments: 33 (3 by maintainers)

Commits related to this issue

Most upvoted comments

After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.

It doesn’t make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.

image

I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.

To reiterate, the steps are:

  1. Go to the packages page
  2. Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings)
  3. “Add a repository” - in my case, it is jauderho/dockerfiles and enable write access
  4. Enable “Inherit access from source repository (recommended)”

What’s odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.

I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?

Update: All of my images have successfully built.

For me none of the above worked, after wandering over the internet and github for 13 Hrs! I found out my solution [^1]:

Select the Read and write permissions option for Workflow permissions in the Actions’ settings page of the repository (https://github.com/<username>/<repository>/settings/actions). 👍🏻

[^1]: Note: I did this after ensuring steps 3 and 4 of @jauderho’s comment were done.

You can find it in https://github.com/orgs/[XXX]/packages/container/[YYY]/settings

or by going to the package page then “package settings”

For me, that just says “get started with packages”. There’s no option to create a new one or anything.

This issue should seriously be reopened. I’ve been struggling trying to fix this for at least three hours now, and I’m well versed in both Github actions and Docker.

I have no packages… so I can’t access the basically hidden “package settings” link: for people reading this, you must click on any package that is listed if you have any, scroll two thirds down the page and you’ll see on the right a “package settings” link. Screenshot from 2023-06-10 21-36-05

My repo allows for all github actions to be executed.

Screenshot from 2023-06-10 21-37-51

And the GITHUB TOKEN has all permissions Screenshot from 2023-06-10 21-38-06

So yup, here I am unable to figure out why this isn’t working but really, this is a Github UX and Docker action problem.

@filipesilva

on push to a PR

Not related to that issue but check permissions of the GITHUB_TOKEN in your workflow on PR. Afaik it will not work without a pull_request_target event but be very careful with it.


Closing this issue as it seems to be solved.

For anyone who struggles to get to that settings page, two hints:

  1. For organizations link will be https://github.com/orgs/ORG NAME/packages/container/PACKAGE NAME/settings
  2. You can get from UI there: image

I think this 403 has to do with how the github packages registry resolves. Using any varient of ghcr.io just in the tags or as the registry property doesn’t work. Using the registry: docker.pkg.github.com how ever solves my issues.

I’m also seeing this problem on push to a PR, but not on push for tags. So it might be somehow related to the metadata, which is different between PR push and tag?

In my case these were the tags for a build that didn’t work:

Docker tags
  ghcr.io/athensresearch/athens-server:pr-1684

but these tags worked:

Docker tags
  ghcr.io/athensresearch/athens-server:mock-v0.1
  ghcr.io/athensresearch/athens-server:latest

In my case the image did not exist at all to begin with, so I couldn’t follow the steps in https://github.com/docker/build-push-action/issues/463#issuecomment-926229761.

For anyone stumbling into this error and have the same issue as @randomairborne, make sure that your personal token has write:packages included in scope. That was the fix for me.

image

I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.

To reiterate, the steps are:

  1. Go to the packages page
  2. Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings)
  3. “Add a repository” - in my case, it is jauderho/dockerfiles and enable write access
  4. Enable “Inherit access from source repository (recommended)”

What’s odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.

I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?

Update: All of my images have successfully built.

Apparently this error also triggers when you set an “invalid” tag.

As stated here, periods in tags are allowed. Screenshot 2022-12-16 at 10 52 51

The workflow step looked something like this:

- name: Build and push
  uses: docker/build-push-action@v1
  with:
    push: true
    load: false
    file: path/to/Dockerfile
    context: .
    tags: ${{ env.DOCKER_TAGS }}

Where ${{ env.DOCKER_TAGS }} was something like this.is.my.tag.

How is this tag invalid you ask? Well it seems like periods are not allowed in this case because this also triggered the above mentioned error. In my opinion this error is rather misleading!

I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.

To reiterate, the steps are:

1. Go to the packages page

2. Pick a package and set the following (eg https://github.com/users/jauderho/packages/container/dnscrypt-proxy/settings)

3. "Add a repository" - in my case, it is jauderho/dockerfiles and enable write access

4. Enable "Inherit access from source repository (recommended)"

What’s odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.

I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?

Update: All of my images have successfully built.

I’m feeling dumb. Where is the packages page ?

Switching to a PAT worked for me. I used this action and redhat-actions/buildah-build and redhat-actions/push-to-registry and both resulted in a 403. Even with adding the repo to the package, granting explicit access, etc.

Once I returned to a PAT it worked flawlessly.