buildx: Qemu 5.2 regression causes arm64 detection flakiness (Reverted)
I’m using buildx to do multiplatform builds in our CI environment and recently we started getting some odd errors. I’m able to reproduce this locally with this simple Dockerfile:
FROM amazonlinux
RUN yum -y update
I set up my multiplatform build this way:
docker buildx create --platform linux/amd64,linux/arm64 --use
Then run the build command. When I run it in multi-platform, I get an error:
[vagrant@vagrant temp]$ docker buildx build --pull --no-cache --platform linux/amd64,linux/arm64 -t motus/testbuild .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.8s (8/8) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 31B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [linux/arm64 internal] load metadata for docker.io/library/amazonlinux:latest 0.1s
=> [linux/amd64 internal] load metadata for docker.io/library/amazonlinux:latest 0.1s
=> CACHED [linux/amd64 1/2] FROM docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152 0.0s
=> => resolve docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152 0.0s
=> CACHED [linux/arm64 1/2] FROM docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152 0.0s
=> => resolve docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152 0.0s
=> CANCELED [linux/amd64 2/2] RUN yum -y update 0.6s
=> ERROR [linux/arm64 2/2] RUN yum -y update 0.5s
------
> [linux/arm64 2/2] RUN yum -y update:
#8 0.370 /usr/bin/python: can't open file 'yum': [Errno 2] No such file or directory
------
Dockerfile:3
--------------------
1 | FROM amazonlinux
2 |
3 | >>> RUN yum -y update
4 |
5 |
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c yum -y update]: exit code: 2
but when I run it for just arm64 it works:
[vagrant@vagrant temp]$ docker buildx build --pull --no-cache --platform linux/arm64 -t motus/testbuild .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 12.6s (5/5) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 31B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/amazonlinux:latest 0.3s
=> CACHED [1/2] FROM docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152 0.0s
=> => resolve docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152 0.0s
=> [2/2] RUN yum -y update 12.2s
Any thoughts on this? I have the qemu aarch64 emulator installed.
Here’s my docker info:
[vagrant@vagrant temp]$ docker system info
Client:
Debug Mode: false
Plugins:
buildx: Build with BuildKit (Docker Inc., v0.5.1)
Server:
Containers: 63
Running: 1
Paused: 0
Stopped: 62
Images: 148
Server Version: 19.03.13-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0 (expected: fec3683)
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.214-160.339.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 22.98GiB
Name: vagrant
ID: MDVE:KCFM:HGLC:XUYO:OXRX:G4SG:Y74M:7GAM:QE3H:K364:YFYL:SH3X
Docker Root Dir: /vagrant-storage/docker
Debug Mode: false
Username: srankin
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 26 (12 by maintainers)
Commits related to this issue
- Adding flag to use buildx in build image core (#4419) I changed the way API proxy builds image in previous PR. I added a flag to turn on buildx to do cross build. I forgot to toggle that tag for bui... — committed to Azure/iotedge by huguesBouvier 3 years ago
- revert to 5.0.1 regression docker/buildx#542 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> — committed to tonistiigi/binfmt by tonistiigi 3 years ago
- Update tests - Pin version for multiarch/qemu-user-static due to QEMU 5.2 regression Ref: https://github.com/docker/buildx/issues/542 — committed to hwdsl2/docker-ipsec-vpn-server by hwdsl2 3 years ago
- Attempt to fix buildx/multiarch build flakiness Hit this multiarch build issue on a seperate PR https://github.com/envoyproxy/envoy/pull/15204#issuecomment-790880847 Fixes https://github.com/envoypr... — committed to arkodg/envoy by arkodg 3 years ago
- Attempt to fix buildx/multiarch build flakiness (#15386) Hit this multiarch build issue on a seperate PR https://github.com/envoyproxy/envoy/pull/15204#issuecomment-790880847 Fixes https://github... — committed to envoyproxy/envoy by arkodg 3 years ago
- Attempt to fix buildx/multiarch build flakiness (#15386) Hit this multiarch build issue on a seperate PR https://github.com/envoyproxy/envoy/pull/15204#issuecomment-790880847 Fixes https://github... — committed to auni53/envoy by arkodg 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to kinarashah/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to rancher/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to superseb/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to superseb/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to pennyscissors/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to pennyscissors/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to galal-hussein/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to galal-hussein/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to pennyscissors/ingress-nginx by kinarashah 3 years ago
- Attempt to fix buildx/multiarch build flakiness (#15386) Hit this multiarch build issue on a seperate PR https://github.com/envoyproxy/envoy/pull/15204#issuecomment-790880847 Fixes https://github... — committed to rexengineering/istio-envoy by arkodg 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to pennyscissors/ingress-nginx by kinarashah 3 years ago
- Fix flaky arm64 emulator issue https://github.com/docker/buildx/issues/542 — committed to pennyscissors/ingress-nginx by kinarashah 3 years ago
@tonistiigi Cannot repro anymore since v6.0.0-rc0. Seems https://github.com/qemu/qemu/commit/0266e8e3b3981b492e82be20bb97e8ed9792ed00 solves the issue.
@jlesage @sdwr98 Can you make some tests on your side with
crazymax/binfmt:v6.0.0-rc4
?I did a couple of builds with
crazymax/binfmt:v6.0.0-rc4
and didn’t see the issue, they all succeeded.We do have QEMU installed via this method:
Here’s the other info you requested: