moby: [docker.pkg.github.com] Docker pull is not working on 20.10 RC1

Installed latest docker 20.10 RC1 and trying to pull images from private registry is not working. It’s working for certain private registry server and not working for others (not sure what logs will help you out more)

Steps to reproduce the issue:

docker login https://docker.pkg.github.com -u name -p token
docker pull docker.pkg.github.com/github/xxxx/backend:latest

Describe the results you received:

docker: Error response from daemon: manifest for docker.pkg.github.com/github/xxxx/backend:latest not found: manifest unknown: Docker image reference backend:sha256:d3exxxxxxxxx5 not found under repo "github".

Describe the results you expected:

same image pull with docker 19.x works without an issue.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

 Version:           20.10.0-rc1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        5cc2396
 Built:             Tue Nov 17 22:51:53 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0-rc1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       131bf7e
  Built:            Tue Nov 17 22:50:10 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.1
  GitCommit:        c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 20.10.0-rc1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.8.0-7630-generic
 Operating System: Pop!_OS 20.10
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 62.79GiB
 Name: pop-os
 ID: LNMV:YDEA:GDHT:NAN3:BG2Z:ISUJ:JVDA:ZTYN:B5XL:VCFI:FQYG:QWHN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio weight support
WARNING: No blkio weight_device support

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (10 by maintainers)

Most upvoted comments

The change in RC1 related to this is https://github.com/moby/moby/pull/41607

The question is whether it accidentally worked before or not. That’s what we are trying to figure out now.

Yes, the docker.pkg.github.com registry is known to not be compliant with the distribution specification (https://github.com/opencontainers/distribution-spec), and only implements parts of the specification. Previously, it “mostly” worked, because it implemented just enough of the specification to make docker pull work, but it failed for many other cases either with docker (https://github.com/moby/moby/issues/40403, https://github.com/moby/moby/issues/34153), containerd (https://github.com/containerd/containerd/issues/3291), and many other runtimes/projects (see the issues linked to/from https://github.com/containerd/containerd/issues/3291 - you may have to “unfold” some comments, as the GitHub UI is hiding them).

IIRC, the missing part is that docker.pkg.github.com does not support pull by digests, which previously was not hit by default on docker pull, but now is (since https://github.com/moby/moby/pull/41607), causing the pull to fail.

thaJeztah on Nov 25, 2020

Thanks. Looks like we need a fallback in e4cf1c733677ff77e51101e2de542373b449b471 for registries that aren’t doing pull by digest.

I’m not sure that makes sense – this is a pretty special case, and I think there are good reasons this wasn’t fixed in containerd (in https://github.com/containerd/containerd/issues/3291).

There are essentially two relevant specifications here – the original Docker Registry API spec and the OCI distribution spec. Both are pretty clear that GET on a digest to the /manifests/ endpoint should succeed, and IMO having HEAD return sucessfully with a valid Docker-Content-Digest is supporting evidence that this should be supported.

What I would suggest instead is that if it is too difficult for docker.pkg.github.com to support pull-by-digest, it should probably be returning a 405 on HEAD requests for manifests (which would transparently solve this issue since we would then fallback to GET instead, and likely fix containerd’s resolver/fetcher too).

Given that https://github.com/containerd/containerd/issues/3291 wasn’t actually resolved, users will see similar failures if they try to use those same images in a Kubernetes environment with containerd as the runtime, so I think there’s really still a pretty good incentive to do something here, even if it isn’t as optimistic as “adding support for pull-by-digest” – just responding that HEAD isn’t supported will solve a large percentage of cases and feels like it should be lower effort.

tianon on Nov 18, 2020

@tianon I don’t know if it’s already tracked, but the issue tracker at https://gitlab.com/gitlab-org/gitlab/-/issues would be the place to report this and request a fix/improvement.

tmaczukin on Dec 2, 2020

https://github.com/moby/moby/pull/41749 is not included in RC2, master should work

AkihiroSuda on Dec 7, 2020

https://gitlab.com/gitlab-org/gitlab/-/issues/290388 seemed like a really closely related discussion (since this would affect any pulling via containerd from https://docs.gitlab.com/ee/user/packages/dependency_proxy/), so I’ve left a comment there. 👍

(Thank you!!)

tianon on Dec 3, 2020

Is it best to throw out warning for older spec registry in 20.10 and break it later?

Well, that’s where this gets dicey - that registry only incidentally worked on 19.03. It isn’t using an older spec, it’s noncompliant with either relevant spec, as-is. Even detecting it is going to amount to a special case in the Docker code for this one specific registry, which doesn’t make sense IMO given this registry is failing because it doesn’t implement the spec correctly (and this registry is supposed to be phased out in usage anyhow, from what I understand).

tianon on Nov 19, 2020

Thanks. Looks like we need a fallback in https://github.com/moby/moby/pull/41607/commits/e4cf1c733677ff77e51101e2de542373b449b471 for registries that aren’t doing pull by digest.

This is, I expect, the same issue as described here: https://github.com/containerd/containerd/issues/3291

This should work with ghcr.

cpuguy83 on Nov 18, 2020