containerd: containerd can't pull image from Github Docker Package Registry

@clarkbw wondering if there’s any progress on GitHub’s side on this issue?

👋 Ability to pull images by digest is in our backlog for this quarter. Apologies for the delay.

@clarkbw any news? I really think you’re underestimating the seriousness of this issue. Simply nobody who’s using Kubernetes with containerd (which is an uncontrollable combination in many cases) can use Github container registry.

@clarkbw do you have any details on planning?

Yes, we’re actively working on a solution. Give my team until the end of April and we’ll have something for you all to test.

We can close this issue out now. GHCR was (beta) released today.

I ran this again.

export DOMAIN="ghcr.io"
for v in "list.v2+json" "v2+json"; do
echo "\n$v\n"
curl --head -H "Authorization: Bearer $(echo $PAT | base64)" \
       -H "Accept: application/vnd.docker.distribution.manifest.$v" \
      https://$DOMAIN/v2/clarkbw/docker-test%2Ftest/manifests/latest;
done
echo "\nv1+prettyjws\n"
curl -i -H "Authorization: Bearer $(echo $PAT | base64)" \
       -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" \
      https://$DOMAIN/v2/clarkbw/docker-test%2Ftest/manifests/latest;

• List support is not yet implemented, should land next week.
• v2+json HEAD works as expected
• v1+prettyjws / v1 only returns the v2 manifest so I’ll look into this

list.v2+json

HTTP/1.1 200 OK
Date: Tue, 01 Sep 2020 10:52:02 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 938
Docker-Content-Digest: sha256:7b4bfd2c49c4fe994bcc80c47a7111a5c4d719e4c569d03bc7133aba25efcd9d
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:7b4bfd2c49c4fe994bcc80c47a7111a5c4d719e4c569d03bc7133aba25efcd9d"
Vary: Accept-Encoding
X-GitHub-Request-Id: EFB8:1F72:6C72:173E9:5F4E27D1


v2+json

HTTP/1.1 200 OK
Date: Tue, 01 Sep 2020 10:52:03 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 938
Docker-Content-Digest: sha256:7b4bfd2c49c4fe994bcc80c47a7111a5c4d719e4c569d03bc7133aba25efcd9d
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:7b4bfd2c49c4fe994bcc80c47a7111a5c4d719e4c569d03bc7133aba25efcd9d"
Vary: Accept-Encoding
X-GitHub-Request-Id: EFB9:6B5F:9839:20F59:5F4E27D2


v1+prettyjws

HTTP/1.1 200 OK
Date: Tue, 01 Sep 2020 10:52:03 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 938
Docker-Content-Digest: sha256:7b4bfd2c49c4fe994bcc80c47a7111a5c4d719e4c569d03bc7133aba25efcd9d
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:7b4bfd2c49c4fe994bcc80c47a7111a5c4d719e4c569d03bc7133aba25efcd9d"
Vary: Accept-Encoding
X-GitHub-Request-Id: EFBA:8C1C:18E0:679C:5F4E27D3

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 5161,
      "digest": "sha256:9edabe03773d1e6ab8c7589baf6837e21ad54e266184511a3df48d29700ff123"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 112,
         "digest": "sha256:b185757d6eb6ee9bd248ccec8899c7f40414136ff6976b7f1b2ba468507b20eb"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 112,
         "digest": "sha256:e1f48cedab3f8273796a8fc048cc8b38e7a0237bbb8c7dbffd76746f3266a8ef"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 112,
         "digest": "sha256:dcbe8992ae1159aee4676ff9dc97b2769c70ace099b5288c5ae7ae9915cd821c"
      }
   ]
}

I ❤️ GitHub but this really makes me question the integrity and professionalism of this service.

Right, thanks; this work is being done by humans with integrity and professionalism. I’m sorry we’ve let you down.

We are rolling out the feature flag to a number of people. I’m sorry if you’ve emailed me and haven’t gotten into the beta yet. Still more coming this week.

Thanks for checking in!

Dates slipped a little due to extenuating circumstances.

I’ll send a message to the maintainer group in mid May with the details for access and feedback.

Should be a public announcement by end of May.

The issue of direct sha access is with our current Docker offering. I’ve connected nearly everyone who reached out from this thread to the fix we have running. Please reach out to me clarkbw@github.com and I can get you setup as well. Hopefully soon we’ll be able to have a public offering here.

We are beginning a private beta this week. Email me, clarkbw@github.com to gain access. We had planned to be in a public beta by this point but 2020, oh she had other plans.

This is absurd.

I ❤️ GitHub but this really makes me question the integrity and professionalism of this service. I’m currently paying for a product that doesn’t even work. It’s been a month since the private beta and literally nothing notably has happened. This is a serious issue… Probably moving to Docker Hub this week… 💔

We’ll have more public news in September. Not far now.

Sorry to be annoying, but do you have any update to share @clarkbw as it has been a couple of weeks since the last update. Getting tripped up on this again after coming back hoping this would’ve been resolved by now.

No credentials are provided until a 401 is received from a registry informing containerd what type of Authorization is expected. This could be basic or bearer.

alright, that was the 🔑 , I know how to fix this in GitHub Docker Registry 😄

I’ll start working on a fix and comment on this thread when I have the fix deployed to production.

Thank you so much for trying out GPR and giving us valuable feedback ❤️

We are beginning a private beta this week. Email me, clarkbw@github.com to gain access. We had planned to be in a public beta by this point but 2020, oh she had other plans.

The private beta is rolling out to a number of users. This is a phased rollout over time so you may only get your instructions over the next couple weeks. Thanks!

We’ll be opening up to the maintainers group very soon and then we’ll have a Beta to share more broadly. Sorry for the delays, I appreciate your patience here.

I believe I see this error as well:

$ ctr --debug images pull docker.pkg.github.com/owner/repo/image:tag
DEBU[0000] fetching                                      image="docker.pkg.github.com/owner/repo/image:tag"
DEBU[0000] resolving                                    
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] fetch response received                       response.headers=map[Content-Length:[52] Content-Type:[application/json] Date:[Mon, 02 Dec 2019 14:06:03 GMT] X-Github-Request-Id:[DF8A:674C:1C4B:1D46C:5DE51A4B]] status="405 Method Not Allowed" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] User-Agent:[containerd/1.2.10-0ubuntu1]] request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] fetch response received                       response.headers=map[Content-Length:[84] Content-Security-Policy:[default-src 'none';] Content-Type:[text/plain; charset=utf-8] Date:[Mon, 02 Dec 2019 14:06:03 GMT] Server:[GitHub Registry] Strict-Transport-Security:[max-age=31536000;] Www-Authenticate:[Basic realm="GitHub Package Registry"] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Github-Request-Id:[DF8A:674C:1C4C:1D46D:5DE51A4B] X-Xss-Protection:[1; mode=block]] status="401 Unauthorized" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[0000] Unauthorized                                  header="Basic realm="GitHub Package Registry""
ctr: failed to resolve reference "docker.pkg.github.com/owner/repo/image:tag": unexpected status code https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag: 401 Unauthorized

When I add the --user flag the error is different, yet still present:

$ k3s ctr --debug images pull --user riker09:$GITHUB_TOKEN docker.pkg.github.com/owner/repo/image:tag
DEBU[2019-12-02T15:29:00.411982181+01:00] fetching                                      image="docker.pkg.github.com/owner/repo/image:tag"
DEBU[2019-12-02T15:29:00.414453549+01:00] resolving                                     host=docker.pkg.github.com
DEBU[2019-12-02T15:29:00.414504221+01:00] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=HEAD url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:00.824424265+01:00] fetch response received                       host=docker.pkg.github.com response.header.content-length=52 response.header.content-type=application/json response.header.date="Mon, 02 Dec 2019 14:29:00 GMT" response.header.x-github-request-id="B478:1FE2:16731:10F61A:5DE51FAC" response.status="405 Method Not Allowed" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:00.824488827+01:00] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.128049683+01:00] fetch response received                       host=docker.pkg.github.com response.header.content-length=84 response.header.content-security-policy="default-src 'none';" response.header.content-type="text/plain; charset=utf-8" response.header.date="Mon, 02 Dec 2019 14:29:00 GMT" response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.www-authenticate="Basic realm=\"GitHub Package Registry\"" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B478:1FE2:16732:10F61B:5DE51FAC" response.header.x-xss-protection="1; mode=block" response.status="401 Unauthorized" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.128144926+01:00] Unauthorized                                  header="Basic realm=\"GitHub Package Registry\"" host=docker.pkg.github.com
DEBU[2019-12-02T15:29:01.128256245+01:00] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.570887786+01:00] fetch response received                       host=docker.pkg.github.com response.header.content-length=2000 response.header.content-security-policy="default-src 'none';" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 02 Dec 2019 14:29:01 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B48A:3EF6:D341:A5D71:5DE51FAD" response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:01.570999503+01:00] no Docker-Content-Digest header, fetching manifest instead  host=docker.pkg.github.com
DEBU[2019-12-02T15:29:01.571038435+01:00] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:02.152267193+01:00] fetch response received                       host=docker.pkg.github.com response.header.content-length=2000 response.header.content-security-policy="default-src 'none';" response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 02 Dec 2019 14:29:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B494:36BD:B6ED:94D55:5DE51FAD" response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/tag"
DEBU[2019-12-02T15:29:02.152555755+01:00] resolved                                      desc.digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" host=docker.pkg.github.com
DEBU[2019-12-02T15:29:02.152779654+01:00] fetch                                         digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json size=2000
DEBU[2019-12-02T15:29:02.163557378+01:00] do request                                    digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
DEBU[2019-12-02T15:29:02.423060126+01:00] fetch response received                       digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=206 response.header.content-security-policy="default-src 'none';" response.header.content-type="text/plain; charset=utf-8" response.header.date="Mon, 02 Dec 2019 14:29:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B494:36BD:B6EE:94D57:5DE51FAE" response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/manifests/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
DEBU[2019-12-02T15:29:02.423468150+01:00] do request                                    digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.3.0-k3s.4 request.method=GET size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/blobs/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
DEBU[2019-12-02T15:29:02.971197798+01:00] fetch response received                       digest="sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=209 response.header.content-security-policy="default-src 'none';" response.header.content-type="text/plain; charset=utf-8" response.header.date="Mon, 02 Dec 2019 14:29:02 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server="GitHub Registry" response.header.strict-transport-security="max-age=31536000;" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.header.x-github-request-id="B4A4:16DD:E38E:A3798:5DE51FAE" response.header.x-xss-protection="1; mode=block" response.status="404 Not Found" size=2000 url="https://docker.pkg.github.com/v2/owner/repo/image/blobs/sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171"
ctr: failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:8ebd12281a627f75b2d2ceb1224e2c1c0a8bfc081e95a2bfcb8077db7a4ab171 (application/vnd.docker.distribution.manifest.v2+json) from remote: not found

Please note: I have replaced the real image name with owner/repo/image:tag.

@Phanatic any updates on this?

Pushed out all the latest invites. Was out on holiday last week so apologies for the delay.

Please send feedback, good or bad. Thanks!

the containerd image pull flow doesn’t provide credentials on first try

No credentials are provided until a 401 is received from a registry informing containerd what type of Authorization is expected. This could be basic or bearer.

Is there a change required here for GitHub Docker Registry to work with containerd?

containerd does not every contact https://registry-1.docker.io/v2/ as this endpoint provides no purpose to the overall registry flow. This endpoint was originally put in place to distinguish a v2 registry from a v1 registry index server (this has been long deprecated and never supported by containerd). Later this year Docker will no longer contact this endpoint either as v1 registry support has been completely removed in the upcoming version of Docker.

New GHCR docs are up

the new registry, you don’t use a git repository-name

You can continue to namespace images down to the repo with GHCR but it is no longer required. Here are our recommended migration steps

Hello, can anyone provide feedback on the private beta? Does it involve some workaround / changing anything on your side? Or will it just fix your/your organization’s Github and work as expected? Thank you.

@clarkbw I’m still waiting fort the invite 🙏

It has been great for me! I had to change references to the new location, and get my kubernetes regcred working, but the beta itself is working smoothly.

I’m getting the following error when trying to pull with miicrok8s.

Failed to pull image "docker.pkg.github.com/resplendent-data/front-end/frontend": rpc error: code = NotFound desc = failed to pull and unpack image "docker.pkg.github.com/resplendent-data/front-end/frontend:latest": failed to copy: httpReaderSeeker: failed open: content at https://docker.pkg.github.com/v2/resplendent-data/front-end/frontend/manifests/sha256:XXXX not found: not found

Doing a docker pull works fine.

Sorry you got stuck. We are working on a solution and will have news soon.

I don’t think is only related to github registry I also faced it with docker trusted registry.

On Tue, 21 Jan 2020 at 18:45 Alex Ellis notifications@github.com wrote:

Just to be clear, this is being classed as an issue with GitHub’s registry, rather than with containerd. Is there somewhere upstream that we can track it and link back to? Anybody have friends with GitHub engineering?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/containerd/containerd/issues/3291?email_source=notifications&email_token=AA4YEQCKTU73KYBCBXXE7KTQ64YCRA5CNFSM4HOBYNNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQTY3I#issuecomment-576797805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4YEQGIPA4TUNTU5G6ZBMLQ64YCRANCNFSM4HOBYNNA .

Is there at least some workaround to make it work at the moment?

Seeing this error with IBM IKS, containerd, and the docker registry deployed within IKS.

The issue is that the docker registry was backed by an external auth provider. Containerd will not send authentication until it receives a 401 with the method that should be used.

So in the auth server, when authorization header is not present, you need to return a 401 with a header

if (!authorization) {
    return res
      .status(401)
      .set("WWW-Authenticate", "Bearer")
      .end();
}

@Phanatic I would be happy to go over the registry API and how containerd client is using it.

The package registry is only returning 401 on the /v2 “ping” endpoint. This endpoint will be removed from future version of Docker. The expected use of the API is that 401 is returned by any endpoint which requires authorization. When an endpoint returns 404 when no auth is provided, then the client will not know to provide authorization. The expected flow from a client for any resource…if no auth then return 200, 401, or 404 (only when everything is public), if with auth then return 200 or 404. This is because the start of a registry interaction may begin on any resource, as the client may not need all resources.

This is all based on what is defined by https://github.com/opencontainers/distribution-spec/blob/master/spec.md rather than Docker’s current very specific flow.

@clarkbw Do you have a ballpark ETA for public fix?

Expecting more people would be giving GitHub container package registry another chance now that dockerhub updated their usage terms. https://www.docker.com/pricing/retentionfaq

@clarkbw is there any update from the folks at GH? This is still a serious issue…

@clarkbw do you have any details on planning?

I spoke with a PM from GH, seems the issue is on their backlog but it is the number one issue they are seeing with their registry implementation.

Hi, I am trying to use Kubernetes Kind and it seems I am hitting this issue, too . https://github.com/kubernetes-sigs/kind/issues/870

I tried both 1.2 and 1.3 latest released versions and I am still getting the same error.

root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr version
Client:
  Version:  v1.2.10
  Revision: b34a5c8af56e510852c35414db4c1f4fa6172339

Server:
  Version:  v1.2.10
  Revision: b34a5c8af56e510852c35414db4c1f4fa6172339

root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr --debug images pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64
DEBU[0000] fetching                                      image="docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] resolving
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] fetch response received                       response.headers=map[Content-Length:[0] Content-Security-Policy:[default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com] Content-Type:[text/html; charset=utf-8] Date:[Thu, 26 Sep 2019 21:06:27 GMT] Expect-Ct:[max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Status:[406 Not Acceptable] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Backend:[Kubernetes] X-Github-Request-Id:[A5AC:664D:2653:1EA75:5D8D2853] X-Request-Id:[2843882f-0329-45f1-8283-91703c6721f9] X-Runtime:[0.005880] X-Runtime-Rack:[0.014772] X-Xss-Protection:[1; mode=block]] status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
ctr: failed to resolve reference "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": unexpected status code https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64: 406 Not Acceptable

root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr version
Client:
  Version:  v1.3.0-rc.3
  Revision: da66333271f372204aed6b83c2ecf37fa7d9ae2c

Server:
  Version:  v1.3.0-rc.3
  Revision: da66333271f372204aed6b83c2ecf37fa7d9ae2c
  UUID: 53b136dc-1bf4-47d2-b079-70bf34ed10fd


root@ubuntu-s-1vcpu-1gb-sfo2-01:~/bin# ./ctr --debug images pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64
DEBU[0000] fetching                                      image="docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] resolving                                     host=docker.pkg.github.com
DEBU[0000] do request                                    host=docker.pkg.github.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *" request.header.user-agent=containerd/v1.3.0-rc.3 request.method=HEAD url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
DEBU[0000] fetch response received                       host=docker.pkg.github.com response.header.content-length=0 response.header.content-security-policy="default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com" response.header.content-type="text/html; charset=utf-8" response.header.date="Thu, 26 Sep 2019 21:10:36 GMT" response.header.expect-ct="max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"" response.header.referrer-policy="origin-when-cross-origin, strict-origin-when-cross-origin" response.header.server=GitHub.com response.header.status="406 Not Acceptable" response.header.strict-transport-security="max-age=31536000; includeSubdomains; preload" response.header.x-content-type-options=nosniff response.header.x-frame-options=deny response.header.x-github-backend=Kubernetes response.header.x-github-request-id="84A8:17A6:67B2:4C076:5D8D294C" response.header.x-request-id=c01dc0e9-5dc5-4818-b0a0-c9a42915e9d3 response.header.x-runtime=0.007552 response.header.x-runtime-rack=0.018101 response.header.x-xss-protection="1; mode=block" response.status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
ctr: failed to resolve reference "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": unexpected status code [manifests v0.9.0-rc.0-25-g1aa27c95_linux_amd64]: 406 Not Acceptable

We would like to see this issue resolved. This is blocking us from using Github registry.

@Phanatic The manifest fetch by digest doesn’t seem to work for me too. Is it because I’m missing something or this is a known issue/feature? Thanks.

Please email me for access!

@clarkbw sorry for hijacking the thread, but this is kind of related to this issue; are public GH package (docker) registries still on the radar?

For anybody having issues with running outdated images with docker swarm, a workaround for me was the Following:

Instead of using a moving tag like :latest, tagging images with the commit sha instead, so for example :latest-5716e43 and then pulling that image. As there is only one image that has the tag I don’t encounter any issues of outdated images on different nodes.

@clarkbw would still be nice to see this asap as the github package registry is pretty useless for Docker if you can’t access images by sha

@clarkbw any news when this will be fixed?

I had this problem and spent 3 days on it, finally stumbled upon this thread that it was is an issue on GitHub’s end.

I don’t understand the details of the specification here. Just from layman’s perspective, all these other docker registry’s work with the docker command. Why can’t containerd just do that docker is doing? I thought the recent Docker releases just uses containerd under the hood.

@clarkbw wondering if there’s any progress on GitHub’s side on this issue?

For my testing use cases, I have “solved” it by moving the images from a private github registry to the one provided by gitlab. But I’m curious to know if GitHub folks are able to solve this for real

@narqo the way I solved was setting up a docker registry in front of the main registry acting as a proxy. Now I can pull/images using containerd.

@tamalsaha This is the same issue that was reported earlier and a registry side fix. @Phanatic for an update.

Thanks, the authentication seems to work now.

The manifest fetch by digest doesn’t seem to work though.

Also I recommend returning the Docker-Content-Digest header on manifest requests to avoid making the client do an extra GET for digest computation.

More generally, fetching by tag is done to resolve to a digest. In this case the registry is used as a trusted source for what that named tag represents. Client may also use external ways to trust a name such as notary or always pinning their deployed images to a manifest digest.

I’m fine closing this one, also allows for Celebration 🥳

@clarkbw could you open the 2 new issues, I would not know how to describe the issue or reproduce.

We can leave this issue open and track, or open 2 new issues in this repo since these things affect containerd users.

Thanks @dfreilich for the tip the change to use ghcr.io and also moving from docker package registry per repo to the new container registry without repo made it work.

My example is now working @clarkbw 🌮 🎉

Events:
  Type    Reason     Age        From                    Message
  ----    ------     ----       ----                    -------
  Normal  Scheduled  <unknown>  default-scheduler       Successfully assigned github-registry/private-reg to 10.186.26.121
  Normal  Pulling    8s         kubelet, 10.186.26.121  Pulling image "ghcr.io/csantanapr/knative-samples_helloworld-go:latest"

I didn’t read that there were changes

The GitHub Container Registry supersedes the existing Packages Docker registry and is optimized to support some of the unique needs of containers.

With the container registry you can:

Store container images within your organization and user account, rather than a repository. Set fine-grained permissions and visibility independent of repository permissions and visibility. Access public container images anonymously.

@csantanapr As per https://docs.github.com/en/packages/getting-started-with-github-container-registry/migrating-to-github-container-registry-for-docker-images#domain-changes, the domain for the registry is ghcr.io, in place of docker.pkg.github.com. Try it with that?

I would like access to test this with faasd please @clarkbw

@clarkbw can you add activengage org, too?

I ❤️ GitHub but this really makes me question the integrity and professionalism of this service.

Right, thanks; this work is being done by humans with integrity and professionalism. I’m sorry we’ve let you down.

We are rolling out the feature flag to a number of people. I’m sorry if you’ve emailed me and haven’t gotten into the beta yet. Still more coming this week.

Awesome; thank you @clarkbw for your personal involvement and supervision on this issue. I’m sure everyone here appreciates that ❤️

Here’s a nasty little workaround for thoses who:

• Don’t mind loosing blue/green deploys until this is resolved
• Don’t mind 10-15 secs app start-up time
• Use docker swarm / docker stack deploys
• Use CI scripts for deployment

In your CI scripts call:

$ docker stack rm {{ your_stack_name }}
$ until [ -z $(docker stack ps {{ your_stack_name }} -q) ]; do sleep 1; done
$ docker stack deploy --with-registry-auth -c docker-compose.yml {{ your_stack_name }}

Basically you ask Docker scheduler to stop all the services under {{ your_stack_name }} orchestrator. A little knack of docker swarm is that docker stack rm will immediately return even if some services are not properly closed chich may cause networking errors when you try to deploy again. That’s why we use a small inline script until [ -z $(docker stack ps {{ your_stack_name }} -q) ]; do sleep 1; done to wait for the proper return.

Hopes it saves a few folks headaches. I guess a similar temporary fix will help you out.

This is quite a frustrating issue, for our apps that MUST use blue/green deploys we bought a private repo to fix the problem.

@jaschaio I tried it with GitHub sha, but it still doesn’t work, once the stack deploy has run, it doesn’t change and gives the same error. Am I missing something here:

I am using it like this

version: "3.3"
services:
  name:
    image: docker.pkg.github.com/user/repo/image_name:${GITHUB_SHA}

PS: I checked, the containers are getting deployed, but the error message is still there. Thanks for your help! Cheers!

A tip here, you can use a Github action which lets you delete docker images from the registry, with the combination of that and this approach I able to deploy and not have a bunch of images lying around.

I got informed by @clarkbw that this is moving on GitHub’s side (but I don’t have an ETA).

@1player very likely the same cause; docker service resolves (and pulls) images by digest, which is what isn’t supported currently by GitHub’s registry

stay tuned

Just to be clear, is this issue being classed as a problem with GitHub’s registry, rather than with containerd? Is there somewhere upstream that we can track it and link back to? Anybody have friends with GitHub engineering?

@cpuguy83 How long has this been on the backlog? I’m concerned that this is a breaking bug, opened almost a year ago and we’re still discussing this. It’s putting me off a little to “commit” to GH.

Is there an ETA on this?

ref: https://docs.docker.com/registry/recipes/mirror/

I try to setup a repro that I can use to debug this further

@Phanatic did you have any luck reproducing it inside a github’s lab? I have the same results as https://github.com/containerd/containerd/issues/3291#issuecomment-560412099 when trying to use github registry from k3s:

$ sudo k3s --version
k3s version v1.17.0-rc.3-k3s.1 (5c374547)
$ sudo k3s ctr --version
ctr github.com/rancher/containerd v1.3.0-k3s.5

@ArpithaDR that is a separate issue and a different registry provider. JFrog’s registry has a different set of issues that they also need to fix upstream, maybe related to #3556

Hello is this issue resolved?

I am getting 406 Not Acceptable when trying to pull from docker.pkg.github.com and I can’t find any reference for this status code anywhere

I have tried both with user creds and without and I got the same output

ctr --debug images pull  docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest
DEBU[2019-09-16T09:20:19.784036734Z] fetching                                      image="docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest"
DEBU[2019-09-16T09:20:19.784162084Z] resolving
DEBU[2019-09-16T09:20:19.784183008Z] do request                                    request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
DEBU[2019-09-16T09:20:20.187236230Z] fetch response received                       response.headers="map[Content-Length:[0] Content-Security-Policy:[default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com] Content-Type:[text/html; charset=utf-8] Date:[Mon, 16 Sep 2019 09:20:20 GMT] Expect-Ct:[max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Status:[406 Not Acceptable] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Backend:[Kubernetes] X-Github-Request-Id:[CF30:6C76:5C7D:6242A:5D7F53D3] X-Request-Id:[6b911397-a1f9-4ad8-8973-d1e1a703ad9d] X-Runtime:[0.004638] X-Runtime-Rack:[0.010905] X-Xss-Protection:[1; mode=block]]" status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
ctr: failed to resolve reference "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest": unexpected status code https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest: 406 Not Acceptable

ctr --debug images pull --user user:pass  docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest
DEBU[2019-09-16T09:22:04.276359468Z] fetching                                      image="docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest"
DEBU[2019-09-16T09:22:04.276496151Z] resolving
DEBU[2019-09-16T09:22:04.276524600Z] do request                                    request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
DEBU[2019-09-16T09:22:04.723342816Z] fetch response received                       response.headers="map[Content-Length:[0] Content-Security-Policy:[default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com] Content-Type:[text/html; charset=utf-8] Date:[Mon, 16 Sep 2019 09:22:04 GMT] Expect-Ct:[max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"] Referrer-Policy:[origin-when-cross-origin, strict-origin-when-cross-origin] Server:[GitHub.com] Status:[406 Not Acceptable] Strict-Transport-Security:[max-age=31536000; includeSubdomains; preload] X-Content-Type-Options:[nosniff] X-Frame-Options:[deny] X-Github-Backend:[Kubernetes] X-Github-Request-Id:[EC7A:54BA:BE46:89229:5D7F543C] X-Request-Id:[c1107b57-bfad-4ad8-96ae-dc0f093908a6] X-Runtime:[0.005667] X-Runtime-Rack:[0.015374] X-Xss-Protection:[1; mode=block]]" status="406 Not Acceptable" url="https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest"
ctr: failed to resolve reference "docker.pkg.github.com/csantanapr/docker/knative-samples_helloworld-go:latest": unexpected status code https://docker.pkg.github.com/v2/csantanapr/docker/knative-samples_helloworld-go/manifests/latest: 406 Not Acceptable

awesome, thanks for the context! I’ll read up on https://github.com/opencontainers/distribution-spec/blob/master/spec.md and try to setup a repro that I can use to debug this further. I’d love to get the gaps identified so we can tackle them all in one go.

Thanks @dmcgowan and @Phanatic for the quick follow up on this.