build-push-action: Random 403 errors when pushing to GHCR
Troubleshooting
Pushing to GHCR seems to randomly resulting in 403s in the last 24 hours. Rerunning the GH action seems to resolve things (partially).
Behaviour
See above.
Failure message : buildx failed with: error: failed to solve: unexpected status: 403 Forbidden
Steps to reproduce this issue
- Kick off GH action to rebuild Docker images
- See https://github.com/jauderho/dockerfiles/actions
- Check one of the recent failed runs. For example, https://github.com/jauderho/dockerfiles/actions/runs/1255324536
Expected behaviour
Builds should complete with no issue.
Actual behaviour
See https://github.com/jauderho/dockerfiles/actions/runs/1255324536
Configuration
- Repository URL (if public): https://github.com/jauderho/dockerfiles
- Build URL (if public): https://github.com/jauderho/dockerfiles/actions/runs/1255324536
https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/tailscale.yml
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 7
- Comments: 33 (3 by maintainers)
Commits related to this issue
- chore: GHCR may not usable now https://github.com/docker/build-push-action/issues/463 — committed to duskmoon314/dev-env by duskmoon314 3 years ago
- Ran into 403 https://github.com/docker/build-push-action/issues/463 — committed to yenlincc/development-containers by deleted user 2 years ago
- chore(ci): Updates the README & GitHub Actions Configuration * Updates the README to direct users to use GitHub Packages for pulling the image. * Updates the GitHub Actions workflow to not logout due... — committed to ndigitals/ols-dockerfiles by timnolte a year ago
- chore(ci): Updates the README & GitHub Actions Configuration (#74) * Updates the README to direct users to use GitHub Packages for pulling the image. * Updates the GitHub Actions workflow to not logo... — committed to ndigitals/ols-dockerfiles by timnolte a year ago
- Use different registry name: https://github.com/docker/build-push-action/issues/463#issuecomment-1563379277 Signed-off-by: Jacob Weinstock <jakobweinstock@gmail.com> — committed to jacobweinstock/dhcrelay by jacobweinstock 7 months ago
After a couple more hours of debugging I fixed it with this under the package settings, or the problem resolved itself.
It doesn’t make sense why this would be needed for one image package in my repo and not the other image packages in the same repo.
For me none of the above worked, after wandering over the internet and github for 13 Hrs! I found out my solution [^1]:
Select the
Read and write permissionsoption forWorkflow permissionsin the Actions’ settings page of the repository (https://github.com/<username>/<repository>/settings/actions).[^1]: Note: I did this after ensuring steps 3 and 4 of @jauderho’s comment were done.
You can find it in https://github.com/orgs/[XXX]/packages/container/[YYY]/settings
or by going to the package page then “package settings”
For me, that just says “get started with packages”. There’s no option to create a new one or anything.
This issue should seriously be reopened. I’ve been struggling trying to fix this for at least three hours now, and I’m well versed in both Github actions and Docker.
I have no packages… so I can’t access the basically hidden “package settings” link: for people reading this, you must click on any package that is listed if you have any, scroll two thirds down the page and you’ll see on the right a “package settings” link.
My repo allows for all github actions to be executed.
And the GITHUB TOKEN has all permissions
So yup, here I am unable to figure out why this isn’t working but really, this is a Github UX and Docker action problem.
@filipesilva
Not related to that issue but check permissions of the
GITHUB_TOKENin your workflow on PR. Afaik it will not work without apull_request_targetevent but be very careful with it.Closing this issue as it seems to be solved.
For anyone who struggles to get to that settings page, two hints:
https://github.com/orgs/ORG NAME/packages/container/PACKAGE NAME/settingsI think this 403 has to do with how the github packages registry resolves. Using any varient of
ghcr.iojust in the tags or as theregistryproperty doesn’t work. Using theregistry: docker.pkg.github.comhow ever solves my issues.I’m also seeing this problem on push to a PR, but not on push for tags. So it might be somehow related to the metadata, which is different between PR push and tag?
In my case these were the tags for a build that didn’t work:
but these tags worked:
In my case the image did not exist at all to begin with, so I couldn’t follow the steps in https://github.com/docker/build-push-action/issues/463#issuecomment-926229761.
For anyone stumbling into this error and have the same issue as @randomairborne, make sure that your personal token has
write:packagesincluded in scope. That was the fix for me.I can confirm that @onedr0p 's solution seems to work. I have now rebuilt two images with no issue and are rerunning jobs on my remaining failed actions.
To reiterate, the steps are:
What’s odd is that as I went through and checked the settings for all of my packages (21!), it seemed some were already set this way and seem to be the ones not having issues.
I wonder if there was some backend maintenance and somehow these settings were lost. Can someone from GitHub investigate and confirm?
Update: All of my images have successfully built.
Apparently this error also triggers when you set an “invalid” tag.
As stated here, periods in tags are allowed.
The workflow step looked something like this:
Where ${{ env.DOCKER_TAGS }} was something like
this.is.my.tag.How is this tag invalid you ask? Well it seems like periods are not allowed in this case because this also triggered the above mentioned error. In my opinion this error is rather misleading!
I’m feeling dumb. Where is the
packages page?Switching to a PAT worked for me. I used this action and
redhat-actions/buildah-buildandredhat-actions/push-to-registryand both resulted in a 403. Even with adding the repo to the package, granting explicit access, etc.Once I returned to a PAT it worked flawlessly.