binfmt: Segmentation fault in latest binfmt image

Minimal Reproduce Steps

  1. Here is the Dockerfile:

    FROM ubuntu:22.04
    
    RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl
    
  2. GitHub Action Job:

   
name: test
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: checkout
        uses: actions/checkout@v4

      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          platforms: linux/amd64,linux/arm64
          #image: tonistiigi/binfmt:qemu-v7.0.0-28

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: login to registry
        uses: docker/login-action@v3
        with:
          registry: docker.io
          username: xxxx
          password: xxxx

      - name: Build and push image
        uses: docker/build-push-action@v6
        with:
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm64
          push: true
          tags: |
            test:latest
  1. The image building will fail with the following logs:
...
#8 76.59 146 added, 0 removed; done.
#8 76.75 Setting up libcurl4:arm64 (7.81.0-1ubuntu1.20) ...
#8 76.76 Setting up curl (7.81.0-1ubuntu1.20) ...
#8 76.77 Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
#8 76.85 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#8 77.41 Segmentation fault (core dumped)
#8 77.49 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#8 77.96 Segmentation fault (core dumped)
#8 77.97 dpkg: error processing package libc-bin (--configure):
#8 77.97  installed libc-bin package post-installation script subprocess returned error exit status 139
#8 77.97 Processing triggers for ca-certificates (20240203~22.04.1) ...
#8 78.24 Updating certificates in /etc/ssl/certs...
#8 96.93 0 added, 0 removed; done.
#8 96.93 Running hooks in /etc/ca-certificates/update.d...
#8 96.96 done.
#8 97.06 Errors were encountered while processing:
#8 97.06  libc-bin
#8 97.14 E: Sub-process /usr/bin/dpkg returned an error code (1)
#8 ERROR: process "/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl" did not complete successfully: exit code: 100
------
 > [linux/arm64 2/2] RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl:
77.97 dpkg: error processing package libc-bin (--configure):
77.97  installed libc-bin package post-installation script subprocess returned error exit status 139
77.97 Processing triggers for ca-certificates (20240203~22.04.1) ...
78.24 Updating certificates in /etc/ssl/certs...
96.93 0 added, 0 removed; done.
96.93 Running hooks in /etc/ca-certificates/update.d...
96.96 done.
97.06 Errors were encountered while processing:
97.06  libc-bin
97.14 E: Sub-process /usr/bin/dpkg returned an error code (1)
------
./Dockerfile:3
--------------------
   1 |     FROM ubuntu:22.04
   2 |     
   3 | >>> RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl
   4 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl" did not complete successfully: exit code: 100
Reference
  builder-a512d6dc-02fb-482e-ae0e-6148721abea8/builder-a512d6dc-02fb-482e-ae0e-6148721abea80/wx4ps25bomstc0f23cechrmij
Check build summary support
  Build summary supported!
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl" did not complete successfully: exit code: 100
...

How to work around

Use tonistiigi/binfmt:qemu-v7.0.0-28. The current latest tonistiigi/binfmt is qemu-v9.2.0-51. I guess there may have some unstable changes in the latest version.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 20
  • Comments: 24 (2 by maintainers)

Commits related to this issue

Most upvoted comments

It looks like every release since 7.0.0-28 until 3 days ago were pre-releases. The way the release workflow works, that means none of the prereleases caused the latest tag to be updated in the image.

Unless I’m missing something, that means people using the latest tag just picked up 2 years worth of changes in one hit, which might make it tricky to track down the root cause 😟

Updated readme to

Image

LMK if more hints for version selection would be needed.

My input to the topic:

  • If it wasn’t clear enough for anyone - this problem has nothing to do with GitHub Actions/Runners. I can reproduce the problem the same way locally or on Azure DevOps.
  • For me issue appeared on ubuntu-22.04 and ubuntu-24.04, but only when cross-compiling to arm64.
  • I don’t exactly know what docker run --install does, but side effects in Docker buildx are there even after image is removed. So for me downgrading to qemu-v7.0.0-28 worked, but ONLY AFTER I cleared the remainings of the previous installation.

So to always make sure that no matter which runner is chosen to run my build, I always run this first:

docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --uninstall linux/arm64
docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --install linux/arm64

So for anyone wondering why downgrading doesn’t work - try uninstalling previous platforms first.

@tonistiigi and @crazy-max I think everyone would appreciate some comments on that. The challenge here is that docker/setup-qemu-action is reasonably popular in the field. It looks like not everyone has a weekly/daily build, but the number of affected workflows will only grow. This overall reveals the opportunistic nature of open source, when many CI pipelines rely on some actions without checking all the details first and hoping that there is some maintenance. This however is a good wakeup call and I’m currently looking for a way to exclude docker/setup-qemu-action from our builds and replace with redhat-actions or similar. However I certainly hope that tonistiigi/binfmt can be still saved. Any comments to highlight the future of this repository would be much appreciated. Even if you decide that there is no capacity to maintain this - clarifying that to the community is very important.

But in my experience there were no issues with compile jobs using QEMUv7 running aarch64 jobs on x86_64.

Using docker/setup-qemu-action@v3 with image: tonistiigi/binfmt:qemu-v7.0.0-28 on ubuntu-22.04 w/ latest Ubuntu kernel was failing for me when I was trying to build arm64 on amd64 machines. (Ref: https://github.com/NVIDIA/cuda-quantum/actions/runs/13294941766/job/37126651103)

Our only workaround available for now is to pin our custom runners to an older Ubuntu kernel, which isn’t a viable long-term solution.

I’ve downgraded Qemu to 7.0.0 so that libc-bin starts working, however, I still get random segfaults from gcc compiling code. Is this related to the issue?

Try running on ARM runners to have a definitive answer - use ubuntu-24.04-arm or ubuntu-22.04-arm.

But in my experience there were no issues with compile jobs using QEMUv7 running aarch64 jobs on x86_64.

ℹ tonistiigi/binfmt:latest is tonistiigi/binfmt:qemu-v9.2.0 since Feb 11, 2025.

I’ve tried tonistiigi/binfmt:qemu-v8.1.5-43 and that’s is crashing on my setup, just like v8.1.5-45, while v7.0.0-28 works.

We are also facing this issue with image “tonistiigi/binfmt:latest” on Ubuntu 24.04 for linux/ppc64le architecture. If we use the version 1 year before i.e. “tonistiigi/binfmt:qemu-v8.1.5-43” then there is no segmentation fault.