kubernetes: k8s.gcr.io/coredns:1.1.3 is not built correctly.

We see this issue in containerd https://github.com/containerd/containerd/issues/2401.

It seems that the size in manifest of k8s.gcr.io/coredns:1.1.3 is wrong.

In the manifest list, the manifest size is 756:

$ curl -v -H 'Host:k8s.gcr.io' -H 'User-Agent:Go-http-client/1.1' -H 'Accept:application/vnd.docker.distribution.manifest.list.v2+json, *' -H 'Accept-Encoding:gzip' 'https://k8s.gcr.io/v2/coredns/manifests/sha256:9a239e9d3060aa5acd3d6e898fe405031594676e15e3b1b471f460eeef462583'  
...
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 756,
         "digest": "sha256:ab8aec91742937058f6b307efbd411ba8b49c5392a757431b00fda4bdd4fa588",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
...

However, the actual manifest size is 950:

$ curl -v -H 'Host:k8s.gcr.io' -H 'User-Agent:Go-http-client/1.1' -H 'Accept:application/vnd.docker.distribution.manifest.v2+json, *' -H 'Accept-Encoding:gzip' 'https://k8s.gcr.io/v2/coredns/manifests/sha256:ab8aec91742937058f6b307efbd411ba8b49c5392a757431b00fda4bdd4fa588'  | wc -c
...
950

Docker doesn’t fail because it is not doing validation. However, containerd does validate it, and fails.

What tools did we use to build this image? /cc @thockin @kubernetes/sig-network-bugs /cc @dmcgowan

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 31 (26 by maintainers)

Most upvoted comments

re-pushed with manifest-tool and it seems good - please confirm

On Fri, Jun 22, 2018 at 12:39 PM Jeff Grafton notifications@github.com wrote:

I’m pretty sure https://github.com/google/containerregistry supports manifest lists properly. I wonder if we can use those for pulling from dockerhub and then pushing to gcr.io.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/65253#issuecomment-399560034, or mute the thread https://github.com/notifications/unsubscribe-auth/AFVgVEviEjFvUvg3VxLBwRPQaPiecLgGks5t_Uf7gaJpZM4UuaqS .

+13

thockin on Jun 22, 2018

/cc @ixdy Discussing this one offline.

BenTheElder on Jun 22, 2018

It works now!

$ crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.0.0-75-ge3d57d2-TEST
RuntimeApiVersion:  v1alpha2
$ crictl pull k8s.gcr.io/coredns:1.1.3
Image is up to date for sha256:b3b94275d97cb24e34af9bb70e8582c312596eaa33716b98b46e0dffdab2f6a4
$ curl -v -H 'Host:k8s.gcr.io' -H 'User-Agent:Go-http-client/1.1' -H 'Accept:application/vnd.docker.distribution.manifest.list.v2+json, *' -H 'Accept-Encoding:gzip' 'https://k8s.gcr.io/v2/coredns/manifests/sha256:db2bf53126ed1c761d5a41f24a1b82a461c85f736ff6e90542e9522be4757848' 
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 950,
         "digest": "sha256:ab8aec91742937058f6b307efbd411ba8b49c5392a757431b00fda4bdd4fa588",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:c8e2c9210cae0dc5ab5bbd4f96a97786494e88d3725dbe44e8ad50223731e06e",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:68c20062c707a4b94092014db3f9f440e357110d0bca97dc8064488fc081ffd3",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:96eaf2c9f4979d91ded1932b8ea7c47a1fdcb30457990b63476c10b9dd7d9786",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 946,
         "digest": "sha256:a6f64e1af34ada82c3837e641fcc50f5fbfd9969c2b0f37ce5b78ba14b14884a",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Random-Liu on Jun 22, 2018

Thank you @vielmetti!

guineveresaenger on Jun 22, 2018