containerd: image pull fails to retrieve k8s.gcr.io/coredns:1.1.3
Output of containerd --version
:
containerd github.com/containerd/containerd v1.1.1-rc.0 395068d2b7256518259816ae19e45824b15da071
$ crictl --version
crictl version 1.0.0-beta.1
The k8s.gcr.io/coredns:1.1.3
image fails to pull
$ sudo crictl pull k8s.gcr.io/coredns:1.1.3
FATA[0001] pulling image failed: rpc error: code = Unknown desc = failed to pull image "k8s.gcr.io/coredns:1.1.3": failed commit on ref "manifest-sha256:a6f64e1af34ada82c3837e641fcc50f5fbfd9969c2b0f37ce5b78ba14b14884a": "manifest-sha256:a6f64e1af34ada82c3837e641fcc50f5fbfd9969c2b0f37ce5b78ba14b14884a" failed size validation: 946 != 752
The image on docker is fine
$ docker pull k8s.gcr.io/coredns:1.1.3
1.1.3: Pulling from coredns
88286f41530e: Pull complete
f7a3e79b147d: Pull complete
8d6991e77daf: Pull complete
Digest: sha256:9a239e9d3060aa5acd3d6e898fe405031594676e15e3b1b471f460eeef462583
Status: Downloaded newer image for k8s.gcr.io/coredns:1.1.3
And crictl can pull the docker version of core-dns ok too
$ sudo crictl pull coredns/coredns:1.1.3
Image is update to date for sha256:b3b94275d97cb24e34af9bb70e8582c312596eaa33716b98b46e0dffdab2f6a4
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 37 (21 by maintainers)
It works now!
from @thockin :
re-pushed with manifest-tool and it seems good - please confirm
@NeilW , @Random-Liu : can you make a try now ?
As far as I can tell, the
docker manifest
command is fine. The error comes fromdocker/distribution
, packageschema2
: https://github.com/docker/distribution/blob/749f6afb4572201e3c37325d0ffedb6f32be8950/manifest/schema2/manifest.go#L92-L93 And well, funny thing, it is NOT canonical, since it is blindly copied from a byte array which might store indented or non-indented JSON: https://github.com/docker/distribution/blob/749f6afb4572201e3c37325d0ffedb6f32be8950/manifest/schema2/manifest.go#L109-L111Here’s a suggested fix to ensure it’s canonical: https://github.com/flx42/cli/commit/8d54e7c9445004bffa38ac798cf984bb243c2d73 (yes, I know this is not the right repo to submit this change, but it’s easier to test).
The right size is whatever the original content size is as returned by the registry. Formatted/compacted does not matter. The problem is somewhere in the manifest list building code it reformatted and lost the original content, producing the wrong size. After a manifest list is built, only that manifest is pushed up, referencing the original manifests which it got from the registry.
Docker is not validating the manifest sizes on pull, only the digest. However Docker will soon switch over the containerd code which will cause these size mismatches to fail validation in Docker.