containerd: image pull fails to retrieve k8s.gcr.io/coredns:1.1.3

Output of containerd --version:

containerd github.com/containerd/containerd v1.1.1-rc.0 395068d2b7256518259816ae19e45824b15da071

$ crictl --version
crictl version 1.0.0-beta.1

The k8s.gcr.io/coredns:1.1.3 image fails to pull

$ sudo crictl pull k8s.gcr.io/coredns:1.1.3
FATA[0001] pulling image failed: rpc error: code = Unknown desc = failed to pull image "k8s.gcr.io/coredns:1.1.3": failed commit on ref "manifest-sha256:a6f64e1af34ada82c3837e641fcc50f5fbfd9969c2b0f37ce5b78ba14b14884a": "manifest-sha256:a6f64e1af34ada82c3837e641fcc50f5fbfd9969c2b0f37ce5b78ba14b14884a" failed size validation: 946 != 752

The image on docker is fine

$ docker pull k8s.gcr.io/coredns:1.1.3
1.1.3: Pulling from coredns
88286f41530e: Pull complete 
f7a3e79b147d: Pull complete 
8d6991e77daf: Pull complete 
Digest: sha256:9a239e9d3060aa5acd3d6e898fe405031594676e15e3b1b471f460eeef462583
Status: Downloaded newer image for k8s.gcr.io/coredns:1.1.3

And crictl can pull the docker version of core-dns ok too

$ sudo crictl pull coredns/coredns:1.1.3
Image is update to date for sha256:b3b94275d97cb24e34af9bb70e8582c312596eaa33716b98b46e0dffdab2f6a4

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 37 (21 by maintainers)

Most upvoted comments

It works now!

$ crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.0.0-75-ge3d57d2-TEST
RuntimeApiVersion:  v1alpha2
$ crictl pull k8s.gcr.io/coredns:1.1.3
Image is up to date for sha256:b3b94275d97cb24e34af9bb70e8582c312596eaa33716b98b46e0dffdab2f6a4
$ curl -v -H 'Host:k8s.gcr.io' -H 'User-Agent:Go-http-client/1.1' -H 'Accept:application/vnd.docker.distribution.manifest.list.v2+json, *' -H 'Accept-Encoding:gzip' 'https://k8s.gcr.io/v2/coredns/manifests/sha256:db2bf53126ed1c761d5a41f24a1b82a461c85f736ff6e90542e9522be4757848' 
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:ab8aec91742937058f6b307efbd411ba8b49c5392a757431b00fda4bdd4fa588",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:c8e2c9210cae0dc5ab5bbd4f96a97786494e88d3725dbe44e8ad50223731e06e",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:68c20062c707a4b94092014db3f9f440e357110d0bca97dc8064488fc081ffd3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:96eaf2c9f4979d91ded1932b8ea7c47a1fdcb30457990b63476c10b9dd7d9786",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:a6f64e1af34ada82c3837e641fcc50f5fbfd9969c2b0f37ce5b78ba14b14884a",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Random-Liu on Jun 22, 2018

from @thockin : re-pushed with manifest-tool and it seems good - please confirm

@NeilW , @Random-Liu : can you make a try now ?

fturib on Jun 22, 2018

As far as I can tell, the docker manifest command is fine. The error comes from docker/distribution, package schema2: https://github.com/docker/distribution/blob/749f6afb4572201e3c37325d0ffedb6f32be8950/manifest/schema2/manifest.go#L92-L93 And well, funny thing, it is NOT canonical, since it is blindly copied from a byte array which might store indented or non-indented JSON: https://github.com/docker/distribution/blob/749f6afb4572201e3c37325d0ffedb6f32be8950/manifest/schema2/manifest.go#L109-L111

Here’s a suggested fix to ensure it’s canonical: https://github.com/flx42/cli/commit/8d54e7c9445004bffa38ac798cf984bb243c2d73 (yes, I know this is not the right repo to submit this change, but it’s easier to test).

flx42 on Jun 22, 2018

I don’t get it, is compacted or formatted JSON the right value for size here?

The right size is whatever the original content size is as returned by the registry. Formatted/compacted does not matter. The problem is somewhere in the manifest list building code it reformatted and lost the original content, producing the wrong size. After a manifest list is built, only that manifest is pushed up, referencing the original manifests which it got from the registry.

Is docker just ignoring the size validation as it works there?

Docker is not validating the manifest sizes on pull, only the digest. However Docker will soon switch over the containerd code which will cause these size mismatches to fail validation in Docker.

dmcgowan on Jun 21, 2018