calico: quay.io/calico/cni:v3.16.5-arm64 - incorrect architecture in manifest and inside the image?

# docker manifest inspect --verbose quay.io/calico/cni:v3.16.5-arm64
{
        "Ref": "quay.io/calico/cni:v3.16.5-arm64",
        "Descriptor": {
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "digest": "sha256:865ad558ba82230b0d388234b7a1d6ed732e454d0102c0a55890dbc6a7b12456",
                "size": 946,
                "platform": {
                        "architecture": "amd64",
                        "os": "linux"
                }
        },
        "SchemaV2Manifest": {
                "schemaVersion": 2,
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "config": {
                        "mediaType": "application/vnd.docker.container.image.v1+json",
                        "size": 2580,
                        "digest": "sha256:f0a2aae8859671bf87c640e57792e3aa96620c086c8345f9879bd5e5aa165894"
                },
                "layers": [
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 18092,
                                "digest": "sha256:3eb1fafec4d364bb44591106c04bccb36ee6101fa9a637377849d6efcf13230d"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 4064,
                                "digest": "sha256:3bc29d15d8a8bb939e4ee7741d8783ff57b7a7efaa48e3214d4e8c1f9b44e953"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 42446614,
                                "digest": "sha256:4224829040d2f578aa392429f195f88d94f87edaec6f1e36fde7dcf16ccf2286"
                        }
                ]
        }
}

Is it correct that there’s "architecture": "amd64"? And it looks like there’re x86_64 binaries inside. For comparison older versions:

# docker manifest inspect --verbose quay.io/calico/cni:v3.15.3-arm64
{
        "Ref": "quay.io/calico/cni:v3.15.3-arm64",
        "Descriptor": {
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "digest": "sha256:7db5bb10620b3af64b141bd2d3ad959b97995c77d6c7e0707f51a8b5ea784719",
                "size": 1573,
                "platform": {
                        "architecture": "arm64",
                        "os": "linux"
                }
        },
        "SchemaV2Manifest": {
                "schemaVersion": 2,
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "config": {
                        "mediaType": "application/vnd.docker.container.image.v1+json",
                        "size": 3459,
                        "digest": "sha256:17e2092f2b36647c82c9ea183c61c68fbaef8cf70a536b54f8227c597dc0054a"
                },
                "layers": [
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 20340179,
                                "digest": "sha256:007027d142c80b166a004bc7265c04036b80df438ac408f1a947e05c581b418e"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 18119,
                                "digest": "sha256:fd6edb08e6506774935240eee44804bab5e75915dc2b63b79ee597dd63a405f1"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 4061,
                                "digest": "sha256:3b95d38f1578719f095a2e3156db9d9267bc447acb84009737ea4b1794ab8b65"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 30695849,
                                "digest": "sha256:b8b44482720a38da86eb0b54d7dd09c38254fe941c60cdb7f1b5a7f06a394dd7"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 3087,
                                "digest": "sha256:8c7602656f2e814f511c5c463a4f33a44d654a78a01ec3bb43c97ada97239f94"
                        },
                        {
                                "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                                "size": 410,
                                "digest": "sha256:34fcbf8be9e764a3486fea0516d1c28f9bf94547a5a19bf1c566aa91f55bf8c0"
                        }
                ]
        }
}

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 23 (12 by maintainers)

Commits related to this issue

enabling eBPF support in arm64. (#1044) Fix: arm64 dockerfile synced with amd64 -> https://github.com/projectcalico/node/issues/524 Add: A new argument`--platform` in docker build phase to address i... — committed to projectcalico/node by frozenprocess 3 years ago

Most upvoted comments

I tested these images in arm64 and binaries are non working:

quay.io/calico/cni                                     v3.15.2             5dadc388f979f       39.8MB
quay.io/calico/cni                                     v3.15.3             ca5564c06ea04       39.8MB
quay.io/calico/cni                                     v3.16.5             9165569ec2362       46.3MB
quay.io/calico/kube-controllers                        v3.15.2             fbbc4a1a0e98e       22.3MB
quay.io/calico/kube-controllers                        v3.16.5             1120bf0b8b414       22.4MB
quay.io/calico/node                                    v3.15.2             cc7508d4d2d4b       91.1MB
quay.io/calico/node                                    v3.16.5             c1fa37765208c       57.3MB

Finally I got it working with docker.io based ones:

docker.io/calico/cni                                   v3.16.5             f0a2aae885967       42.5MB
docker.io/calico/node                                  v3.16.5             e7826a139b1b7       39.4MB

Hope this helps anybody.

pando85 on Jan 16, 2021

Why is there -arm64 suffix in image name?

I think this is a hangover from before Quay supported multi-arch manifest images. Agree that this should be fixed.

lwr20 on Jun 30, 2021

@caseydavenport yes. --target-platform can fix this issue. https://github.com/projectcalico/node/pull/1044/files#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R269

an image created using the PR

docker inspect rezareza/calico-node:latest | egrep -i arch
        "Architecture": "arm64",

frozenprocess on Aug 17, 2021

One thing I don’t get here. Why is there -arm64 suffix in image name? Part of the point is that you do not want to assume the architecture across your containers. Some may run on heterogeneous node groups across AMD and ARM. Image should be just quay.io/calico/node:3.16.4 and the right image for each architecture should be picked because it’s a multi-architecture image built with manifests for arm64 and amd64.

Isn’t this your expectation too?

aquam8 on Jun 29, 2021

The last working image is 3.15.3, question is what changed between 3.15.3 and 3.16?

mrunge on Jan 7, 2021

@caseydavenport This is not just a display issue. The architect field is indeed incorrectly set. You can also test this with run --platform linux/arm64 on amd64 machines.

> docker run --rm -it  --platform linux/arm64  --entrypoint /bin/sh calico/node:v3.16.6-arm64
/ # uname -m
aarch64

> docker run --rm -it  --platform linux/arm64  --entrypoint /bin/sh calico/cni:v3.16.6-arm64
Unable to find image 'calico/cni:v3.16.6-arm64' locally
v3.16.6-arm64: Pulling from calico/cni
Digest: sha256:afd48a8e96fd6090297badf0cbeda126e3a07172eba4e35034733ea4849a224b
Status: Image is up to date for calico/cni:v3.16.6-arm64
docker: Error response from daemon: image with reference calico/cni:v3.16.6-arm64 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.
See 'docker run --help'.

We got below image pull error on Kubernetes (EC2 c6g node),

failed to unpack image on snapshotter overlayfs: no match for platform in manifest sha256:b2410e2206f1ed88dc10f14a9a3fb396d7f5f4b117351511e9ad961bb7c319e4: not found

We noticed the same issue with calico/typha.

lht on Jan 6, 2021