buildx: Qemu 5.2 regression causes arm64 detection flakiness (Reverted)

I’m using buildx to do multiplatform builds in our CI environment and recently we started getting some odd errors. I’m able to reproduce this locally with this simple Dockerfile:

FROM amazonlinux
RUN yum -y update

I set up my multiplatform build this way:

docker buildx create --platform linux/amd64,linux/arm64 --use

Then run the build command. When I run it in multi-platform, I get an error:

[vagrant@vagrant temp]$ docker buildx build  --pull --no-cache --platform linux/amd64,linux/arm64 -t motus/testbuild .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.8s (8/8) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                      0.0s
 => [linux/arm64 internal] load metadata for docker.io/library/amazonlinux:latest                                                                                                                                                                    0.1s
 => [linux/amd64 internal] load metadata for docker.io/library/amazonlinux:latest                                                                                                                                                                    0.1s
 => CACHED [linux/amd64 1/2] FROM docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152                                                                                                              0.0s
 => => resolve docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152                                                                                                                                 0.0s
 => CACHED [linux/arm64 1/2] FROM docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152                                                                                                              0.0s
 => => resolve docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152                                                                                                                                 0.0s
 => CANCELED [linux/amd64 2/2] RUN yum -y update                                                                                                                                                                                                     0.6s
 => ERROR [linux/arm64 2/2] RUN yum -y update                                                                                                                                                                                                        0.5s
------
 > [linux/arm64 2/2] RUN yum -y update:
#8 0.370 /usr/bin/python: can't open file 'yum': [Errno 2] No such file or directory
------
Dockerfile:3
--------------------
   1 |     FROM amazonlinux
   2 |
   3 | >>> RUN yum -y update
   4 |
   5 |
--------------------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c yum -y update]: exit code: 2

but when I run it for just arm64 it works:

[vagrant@vagrant temp]$ docker buildx build  --pull --no-cache --platform linux/arm64 -t motus/testbuild .
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 12.6s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/amazonlinux:latest                                                                                                                                                                                0.3s
 => CACHED [1/2] FROM docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152                                                                                                                          0.0s
 => => resolve docker.io/library/amazonlinux@sha256:ed6a24ee79bb52f2308a20fb20e48b73cfa0b65e89d8a84b6eb68738d4a16152                                                                                                                                 0.0s
 => [2/2] RUN yum -y update                                                                                                                                                                                                                         12.2s

Any thoughts on this? I have the qemu aarch64 emulator installed.

Here’s my docker info:

[vagrant@vagrant temp]$ docker system info
Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1)

Server:
 Containers: 63
  Running: 1
  Paused: 0
  Stopped: 62
 Images: 148
 Server Version: 19.03.13-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0 (expected: fec3683)
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.214-160.339.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 22.98GiB
 Name: vagrant
 ID: MDVE:KCFM:HGLC:XUYO:OXRX:G4SG:Y74M:7GAM:QE3H:K364:YFYL:SH3X
 Docker Root Dir: /vagrant-storage/docker
 Debug Mode: false
 Username: srankin
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 26 (12 by maintainers)

Commits related to this issue

Most upvoted comments

@tonistiigi Cannot repro anymore since v6.0.0-rc0. Seems https://github.com/qemu/qemu/commit/0266e8e3b3981b492e82be20bb97e8ed9792ed00 solves the issue.

@jlesage @sdwr98 Can you make some tests on your side with crazymax/binfmt:v6.0.0-rc4?

I did a couple of builds with crazymax/binfmt:v6.0.0-rc4 and didn’t see the issue, they all succeeded.

We do have QEMU installed via this method:

docker run --privileged --rm tonistiigi/binfmt --install all

Here’s the other info you requested:

[vagrant@vagrant ~]$ docker buildx ls
NAME/NODE         DRIVER/ENDPOINT             STATUS  PLATFORMS
elegant_rhodes *  docker-container
  elegant_rhodes0 unix:///var/run/docker.sock running linux/amd64*, linux/arm64*, linux/386, linux/arm/v7, linux/arm/v6, linux/riscv64, linux/ppc64le, linux/s390x
default           docker
  default         default                     running linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
[vagrant@vagrant ~]$ docker buildx inspect elegant_rhodes
Name:   elegant_rhodes
Driver: docker-container

Nodes:
Name:      elegant_rhodes0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64*, linux/arm64*, linux/386, linux/arm/v7, linux/arm/v6, linux/riscv64, linux/ppc64le, linux/s390x
[vagrant@vagrant ~]$ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/amd64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-arm",
    "qemu-i386",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}