kubernetes: Conformance image v1.18.0 fails on arm64

What happened: gcr.io/google-containers/conformance:v1.18.0 produces an error:

standard_init_linux.go:211: exec user process caused "exec format error"

What you expected to happen: Conformance tests should be launched. Examining image contents indicates executables are of the correct architecture.

How to reproduce it (as minimally and precisely as possible): From aarch64 machine:

docker run --rm -it --entrypoint bash gcr.io/google-containers/conformance:v1.18.0 -c "echo ok"

Anything else we need to know?: Works with v1.17.4:

# docker run --rm -it --entrypoint bash gcr.io/google-containers/conformance:v1.17.4 -c "echo ok"
ok

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0+k3s-5fe1f025", GitCommit:"5fe1f02592397c8a199b898113eb3e798905e918", GitTreeState:"clean", BuildDate:"2020-03-26T23:02:16Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/arm64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0+k3s-5fe1f025", GitCommit:"5fe1f02592397c8a199b898113eb3e798905e918", GitTreeState:"clean", BuildDate:"2020-03-26T23:02:16Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/arm64"}

Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):

Ubuntu 18.04.3 LTS

Kernel (e.g. uname -a):

Linux ip-172-31-17-16 4.15.0-1054-aws #56-Ubuntu SMP Thu Nov 7 16:18:50 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

Install tools:
Network plugin and version (if this is a network-related bug):
Others:

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 24 (17 by maintainers)

Commits related to this issue

Refetch base image for building conformance images The debian base image is actually multi-arch, but docker might not actually pull it if it has a similar one in the registry and that one was for a d... — committed to dirkmueller/kubernetes by dirkmueller 4 years ago

Most upvoted comments

Thanks for taking a look @lubinsz, just to be sure it is not a VM issue I fired up an a1-metal arm64 aws instance. Please note I am using sonobuoy for running tests rather than kubetest, I tried to find if kubetest is using the conformance image for testing and from a quick check maybe not. The error is not in the conformance test itself, but launching the conformance test image on arm64.

The docker --version is 19.03.8, build afacb8b. Tried again with with 1.18.2 and got the same error:

# docker run --rm -it --entrypoint bash gcr.io/google-containers/conformance:v1.18.2 -c "echo ok"
standard_init_linux.go:211: exec user process caused "exec format error"

still works ok for the v1.17 branch:

# docker run --rm -it --entrypoint bash gcr.io/google-containers/conformance:v1.17.5 -c "echo ok"
ok

It looks like the build chain for images has changed:

$ manifest-tool inspect gcr.io/google-containers/conformance-arm64:v1.17.5
gcr.io/google-containers/conformance-arm64:v1.17.5: manifest type: application/vnd.docker.distribution.manifest.v2+json
      Digest: sha256:28c552f98f8782822a0011677ec17a441338e31de7423d760eb8bf01035406bb
Architecture: amd64
          OS: linux
    # Layers: 13
      layer 1: digest = sha256:9eaa7f24d84945e440e52b373346bad450201aa9675ef184d1cf0ad67108e47d
      layer 2: digest = sha256:9be8935edc7b9fa048cbd1a00ed6d460a3fd4b0711b4217878a685c784b2a793
      layer 3: digest = sha256:717a23677156d910c393baecfb7ebead10e4b26a15853847ed4098c5abae2264
      layer 4: digest = sha256:401936c1e26bc5d9fe960f983adf2d5880d665c206477b1228b06ed0a23ccf92
      layer 5: digest = sha256:cc18490b88c366b49d3815bc312d360606f8714d8eaf263767abeb987f37c7b8
      layer 6: digest = sha256:9774a123cd8c073b1e0d842ac4a6e910238d76dd37fa34f3ad943564bd378172
      layer 7: digest = sha256:477d861f083c67b74f28713e954b44b37525e47758b83ef0b08641bf0f9cea8a
      layer 8: digest = sha256:7a4b2319f1ba65ea934130d98c590ce457802623ddbe1ccf985058ee16571f90
      layer 9: digest = sha256:1dd30069b67d20b8cdf6a62c846576fb8d957dd8f1f60d76449f954aa893434a
      layer 10: digest = sha256:493d9e4471f4ebdf9b1b25664928bb57bae15a2853e2e5879a10712ba6833cf9
      layer 11: digest = sha256:6b9d67f9bc67ab6843acb7e30dba31ba2f96b24c66b4195f22372b89355e6ce1
      layer 12: digest = sha256:04e062855c3250a44cda808d8df97dbbd6b2e967cd73a2608c89cc08377895c8
      layer 13: digest = sha256:6e61688fe9eeb5506c9db14a492f753da46652c5145d4f9a29b542e2eba49860

$ manifest-tool inspect gcr.io/google-containers/conformance-arm64:v1.18.2
gcr.io/google-containers/conformance-arm64:v1.18.2: manifest type: application/vnd.docker.distribution.manifest.v2+json
      Digest: sha256:f3182737369625e9a66232eed9851fe4bfe757b7775a5e7d778bcf34f8694699
Architecture: amd64
          OS: linux
    # Layers: 7
      layer 1: digest = sha256:5e35bd43cf7898d036f8515be74d45b2e3abd2a5534fc280de63a9c22dd175bd
      layer 2: digest = sha256:5f8ad54c1ca94ff61cd0ae4a49ee3e2583218460aa112c5e1db2ad470d4fae25
      layer 3: digest = sha256:d6d7389259d48bcef13cadfe180c2446d9d044406d852869d4dc040b4915b44e
      layer 4: digest = sha256:206ef2f1b1a7bdade11d441123a19beb69253de80ae36dc3a835559a85cda798
      layer 5: digest = sha256:330ecaf54ad67f2da66deb9d9abbd267baea8e0cca170580e83031b372bd1143
      layer 6: digest = sha256:e719cefb54f06957b4718738e9e5819fe308c8ca9c727430667b967fe6b11ea6
      layer 7: digest = sha256:a9ab47012e1328866b7fbe731aa6ef0c957927f5ca506a67007017fd134cd7c3

So there are now six less layers of the conformance image for the 1.18 branch.

Please also note how the arm64 images show amd64 as the architecture for both branches. This has been problematic for trying to air-gap arm64 properly and is a systemic issue for many of the projects on gcr (and elsewhere).

erikwilson on Apr 23, 2020

IIRC the fix was reverted due to not working in the release pipeline. I did say that would hurt @cpanato @justaugustus 🙃 https://github.com/kubernetes/kubernetes/pull/92042

BenTheElder on Jun 18, 2020

/reopen

This issue is still unfixed as of today. here is a reproducer that helps doing the reverse test on a x86_64 machine:

> # uname -m
x86_64
> # podman pull gcr.io/google-containers/conformance-arm64:v1.17.7
> # podman run 66652b9ed7f16a9bc848038ddb00159f3e2d22d8c2bb1ba050913309daf101c9  cat /etc/os-release
standard_init_linux.go:211: exec user process caused "exec format error"

So this is an image where the base os is actually arm64. starting with v1.18.0 and still as of today (v1.18.5-rc.0 and v1.19.0-beta.2) the uploaded -arm64 image is actually an amd64 image. There must be a typo somewhere.

testable this way:


$ podman pull gcr.io/google-containers/conformance-arm64:v1.19.0-beta.
...
$ podman run 073f5eae3b09e0c8a221519af5ace0c11c4de707cd3c77616e0d35c058b87b5f  cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

This is supposed to generate the ‘exec file error’, but it doesn’t, which indicates that it is actually an x86_64 image (amd64).

dirkmueller on Jun 18, 2020