binfmt: Intermittent segmentation fault running Node emulated with target arm64, host amd64

Hello, thanks for binfmt.

docker run --privileged --rm tonistiigi/binfmt --version
binfmt/a161c41 qemu/v7.0.0 go/1.18.5

When running

docker buildx build . --platform=arm64 --progress=plain --no-cache

on a Dockerfile which installs node on ubuntu, node --version in the container build intermittently segfaults. I captured it happening with QEMU_STRACE=1 as below.

#14 [11/13] RUN node --version
#14 0.352 1 brk(NULL) = 0x0000005500030000
#14 0.354 1 uname(0x5502830768) = 0
#14 0.355 1 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,0) = -1 errno=2 (No such file or directory)
#14 0.356 1 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
#14 0.356 1 newfstatat(3,"",0x000000550282fd10,0x1000) = 0
#14 0.356 1 mmap(NULL,7675,PROT_READ,MAP_PRIVATE,3,0) = 0x0000005502870000
#14 0.356 1 close(3) = 0
#14 0.357 1 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
#14 0.357 1 read(3,0x282fec0,832) = 832
#14 0.357 1 newfstatat(3,"",0x000000550282fd50,0x1000) = 0
#14 0.358 1 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005502872000
#14 0.358 1 mmap(NULL,1805928,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005502874000
#14 0.358 1 mmap(0x0000005502880000,1740392,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0) = 0x0000005502880000
#14 0.358 1 munmap(0x0000005502874000,49152) = 0
#14 0.358 1 munmap(0x0000005502a29000,15976) = 0
#14 0.358 1 mprotect(0x0000005502a08000,61440,PROT_NONE) = 0
#14 0.359 1 mmap(0x0000005502a17000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x187000) = 0x0000005502a17000
#14 0.359 1 mmap(0x0000005502a1d000,48744,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x0000005502a1d000
#14 0.359 1 close(3) = 0
#14 0.361 1 set_tid_address(365114633616,365114609728,365114633600,365114614160,365114633856,365114633728) = 1
#14 0.361 1 set_robust_list(365114633632,24,365114633632,1,0,365114633680) = -1 errno=38 (Function not implemented)
#14 0.361 1 Unknown syscall 293
#14 0.363 1 mprotect(0x0000005502a17000,16384,PROT_READ) = 0
#14 0.364 1 mprotect(0x000000550002b000,8192,PROT_READ) = 0
#14 0.365 1 mprotect(0x000000550286b000,8192,PROT_READ) = 0
#14 0.365 1 prlimit64(0,3,0,365114362072,365115958040,40) = 0
#14 0.365 1 munmap(0x0000005502870000,7675) = 0
#14 0.366 1 getuid() = 54902
#14 0.366 1 getgid() = 54902
#14 0.366 1 getpid() = 1
#14 0.367 1 rt_sigaction(SIGCHLD,0x0000005502830888,NULL) = 0
#14 0.367 1 geteuid() = 54902
#14 0.367 1 getrandom(365116397872,8,1,365114633408,1,0) = 8
#14 0.368 1 brk(NULL) = 0x0000005500030000
#14 0.368 1 brk(0x0000005500051000) = 0x0000005500051000
#14 0.369 1 getppid() = 0
#14 0.371 1 getcwd(0x5500030390,4096) = 10
#14 0.372 1 geteuid() = 54902
#14 0.372 1 getegid() = 54902
#14 0.372 1 rt_sigaction(SIGINT,NULL,0x00000055028308f0) = 0
#14 0.372 1 rt_sigaction(SIGINT,0x0000005502830858,NULL) = 0
#14 0.372 1 rt_sigaction(SIGQUIT,NULL,0x00000055028308f0) = 0
#14 0.372 1 rt_sigaction(SIGQUIT,0x0000005502830858,NULL) = 0
#14 0.372 1 rt_sigaction(SIGTERM,NULL,0x00000055028308f0) = 0
#14 0.372 1 rt_sigaction(SIGTERM,0x0000005502830858,NULL) = 0
#14 0.375 1 newfstatat(AT_FDCWD,"/opt/node-v20.17.0-linux-arm64/bin//node",0x0000005502830838,0) = 0
#14 0.376 1 rt_sigprocmask(SIG_SETMASK,0x0000005502830838,NULL) = 0
#14 0.376 1 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x00000055028307e0,parent_tidptr=0x0000000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 8
#14 0.376 1 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x00000055028307e0,parent_tidptr=0x0000000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 0
#14 0.376 1 rt_sigprocmask(SIG_SETMASK,0x00000055028306d8,0x0000005502830758) = 0
#14 0.377 8 rt_sigprocmask(SIG_SETMASK,0x00000055028306d8,0x0000005502830758) = 0
#14 0.521 1 wait4(-1,365114361732,0,0,365114633408,0) = 8
#14 0.521 --- SIGCHLD {si_signo=SIGCHLD, si_code=3, si_pid=8, si_uid=54902, si_status=11, si_utime=0, si_stime=0} ---
#14 0.521 1 rt_sigreturn(17,365072415712,1,0,365072404480,0) = -1 errno=513 (Successful exit from sigreturn)
#14 0.525 Segmentation fault (core dumped)
#14 0.525 1 write(2,0x2830808,33) = 33
#14 0.526 1 wait4(-1,365114361732,1,0,365114633408,365114361882) = -1 errno=10 (No child process)
#14 0.529 1 exit_group(139)
#14 ERROR: process "/bin/sh -c node --version" did not complete successfully: exit code: 139
------
 > [11/13] RUN node --version:
0.376 1 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x00000055028307e0,parent_tidptr=0x0000000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 0
0.376 1 rt_sigprocmask(SIG_SETMASK,0x00000055028306d8,0x0000005502830758) = 0
0.377 8 rt_sigprocmask(SIG_SETMASK,0x00000055028306d8,0x0000005502830758) = 0
0.521 1 wait4(-1,365114361732,0,0,365114633408,0) = 8
0.521 --- SIGCHLD {si_signo=SIGCHLD, si_code=3, si_pid=8, si_uid=54902, si_status=11, si_utime=0, si_stime=0} ---
0.521 1 rt_sigreturn(17,365072415712,1,0,365072404480,0) = -1 errno=513 (Successful exit from sigreturn)
0.525 Segmentation fault (core dumped)
0.525 1 write(2,0x2830808,33) = 33
0.526 1 wait4(-1,365114361732,1,0,365114633408,365114361882) = -1 errno=10 (No child process)
0.529 1 exit_group(139)
------

 1 warning found (use docker --debug to expand):
 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 12)
Dockerfile:35
--------------------
  33 |     ENV QEMU_STRACE=1
  34 |
  35 | >>> RUN node --version
  36 |     RUN npm --version
  37 |     RUN yarn --version
--------------------
ERROR: failed to solve: process "/bin/sh -c node --version" did not complete successfully: exit code: 139

For comparison I also provide a log with node --version working, at the same point as where the crash occurred above, but this time working, as it is an intermittent problem.

#14 [11/13] RUN node --version
#14 0.386 1 brk(NULL) = 0x0000005500030000
#14 0.389 1 uname(0x5502830768) = 0
#14 0.391 1 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,0) = -1 errno=2 (No such file or direc
tory)
#14 0.393 1 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
#14 0.393 1 newfstatat(3,"",0x000000550282fd10,0x1000) = 0
#14 0.393 1 mmap(NULL,7675,PROT_READ,MAP_PRIVATE,3,0) = 0x0000005502870000
#14 0.393 1 close(3) = 0
#14 0.394 1 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
#14 0.394 1 read(3,0x282fec0,832) = 832
#14 0.395 1 newfstatat(3,"",0x000000550282fd50,0x1000) = 0
#14 0.395 1 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00000055028
72000
#14 0.396 1 mmap(NULL,1805928,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005502874000
#14 0.396 1 mmap(0x0000005502880000,1740392,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_F
IXED,3,0) = 0x0000005502880000
#14 0.396 1 munmap(0x0000005502874000,49152) = 0
#14 0.396 1 munmap(0x0000005502a29000,15976) = 0
#14 0.396 1 mprotect(0x0000005502a08000,61440,PROT_NONE) = 0
#14 0.397 1 mmap(0x0000005502a17000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FI
XED,3,0x187000) = 0x0000005502a17000
#14 0.397 1 mmap(0x0000005502a1d000,48744,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FI
XED,-1,0) = 0x0000005502a1d000
#14 0.398 1 close(3) = 0
#14 0.402 1 set_tid_address(365114633616,365114609728,365114633600,365114614160,365114633856,36
5114633728) = 1
#14 0.402 1 set_robust_list(365114633632,24,365114633632,1,0,365114633680) = -1 errno=38 (Funct
ion not implemented)
#14 0.402 1 Unknown syscall 293
#14 0.405 1 mprotect(0x0000005502a17000,16384,PROT_READ) = 0
#14 0.405 1 mprotect(0x000000550002b000,8192,PROT_READ) = 0
#14 0.407 1 mprotect(0x000000550286b000,8192,PROT_READ) = 0
#14 0.408 1 prlimit64(0,3,0,365114362072,365115958040,40) = 0
#14 0.409 1 munmap(0x0000005502870000,7675) = 0
#14 0.411 1 getuid() = 54902
#14 0.411 1 getgid() = 54902
#14 0.411 1 getpid() = 1
#14 0.411 1 rt_sigaction(SIGCHLD,0x0000005502830888,NULL) = 0
#14 0.412 1 geteuid() = 54902
#14 0.412 1 getrandom(365116397872,8,1,365114633408,1,0) = 8
#14 0.413 1 brk(NULL) = 0x0000005500030000
#14 0.413 1 brk(0x0000005500051000) = 0x0000005500051000
#14 0.416 1 getppid() = 0
#14 0.418 1 getcwd(0x5500030390,4096) = 10
#14 0.420 1 geteuid() = 54902
#14 0.420 1 getegid() = 54902
#14 0.420 1 rt_sigaction(SIGINT,NULL,0x00000055028308f0) = 0
#14 0.421 1 rt_sigaction(SIGINT,0x0000005502830858,NULL) = 0
#14 0.421 1 rt_sigaction(SIGQUIT,NULL,0x00000055028308f0) = 0
#14 0.421 1 rt_sigaction(SIGQUIT,0x0000005502830858,NULL) = 0
#14 0.421 1 rt_sigaction(SIGTERM,NULL,0x00000055028308f0) = 0
#14 0.421 1 rt_sigaction(SIGTERM,0x0000005502830858,NULL) = 0
#14 0.426 1 newfstatat(AT_FDCWD,"/opt/node-v20.17.0-linux-arm64/bin//node",0x0000005502830838,0
) = 0
#14 0.427 1 rt_sigprocmask(SIG_SETMASK,0x0000005502830838,NULL) = 0
#14 0.427 1 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x00000055028307e0,parent_tidptr=0x0000
000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 8
#14 0.428 1 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x00000055028307e0,parent_tidptr=0x0000
000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 0
#14 0.428 1 rt_sigprocmask(SIG_SETMASK,0x00000055028306d8,0x0000005502830758) = 0
#14 0.428 8 rt_sigprocmask(SIG_SETMASK,0x00000055028306d8,0x0000005502830758) = 0
#14 0.507 8 brk(NULL) = 0x00000000052a8000
#14 0.509 8 uname(0x5500800748) = 0
#14 0.512 8 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,0) = -1 errno=2 (No such file or direc
tory)
#14 0.513 8 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
#14 0.513 8 newfstatat(3,"",0x00000055007ffcf0,0x1000) = 0
#14 0.513 8 mmap(NULL,7675,PROT_READ,MAP_PRIVATE,3,0) = 0x0000005500840000
#14 0.513 8 close(3) = 0
#14 0.514 8 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libdl.so.2",O_RDONLY|O_CLOEXEC) = 3
#14 0.514 8 read(3,0x7ffea0,832) = 832
#14 0.515 8 newfstatat(3,"",0x00000055007ffd30,0x1000) = 0
#14 0.515 8 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00000055008
42000
#14 0.516 8 mmap(NULL,135200,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500844000
#14 0.516 8 mmap(0x0000005500850000,69664,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIX
ED,3,0) = 0x0000005500850000
#14 0.516 8 munmap(0x0000005500844000,49152) = 0
#14 0.517 8 munmap(0x0000005500862000,12320) = 0
#14 0.517 8 mprotect(0x0000005500851000,61440,PROT_NONE) = 0
#14 0.517 8 mmap(0x0000005500860000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIX
ED,3,0) = 0x0000005500860000
#14 0.518 8 close(3) = 0
#14 0.518 8 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libstdc++.so.6",O_RDONLY|O_CLOEXEC) = 3
#14 0.518 8 read(3,0x7ffe80,832) = 832
#14 0.518 8 newfstatat(3,"",0x00000055007ffd10,0x1000) = 0
#14 0.519 8 mmap(NULL,2332704,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500866000
#14 0.519 8 mmap(0x0000005500870000,2267168,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_F
IXED,3,0) = 0x0000005500870000
#14 0.519 8 munmap(0x0000005500866000,40960) = 0
#14 0.519 8 munmap(0x0000005500a9a000,22560) = 0
#14 0.519 8 mprotect(0x0000005500a7a000,61440,PROT_NONE) = 0
#14 0.519 8 mmap(0x0000005500a89000,57344,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FI
XED,3,0x209000) = 0x0000005500a89000
#14 0.519 8 mmap(0x0000005500a97000,10272,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FI
XED,-1,0) = 0x0000005500a97000
#14 0.519 8 close(3) = 0
#14 0.519 8 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libm.so.6",O_RDONLY|O_CLOEXEC) = 3
#14 0.519 8 read(3,0x7ffe60,832) = 832
#14 0.519 8 newfstatat(3,"",0x00000055007ffcf0,0x1000) = 0
#14 0.519 8 mmap(NULL,680048,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500aa0000
#14 0.519 8 mmap(0x0000005500aa0000,614512,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FI
XED,3,0) = 0x0000005500aa0000
#14 0.519 8 munmap(0x0000005500b37000,61552) = 0
#14 0.520 8 mprotect(0x0000005500b26000,61440,PROT_NONE) = 0
#14 0.520 8 mmap(0x0000005500b35000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIX
ED,3,0x85000) = 0x0000005500b35000
#14 0.520 8 close(3) = 0
#14 0.520 8 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libgcc_s.so.1",O_RDONLY|O_CLOEXEC) = 3
#14 0.520 8 read(3,0x7ffe40,832) = 832
#14 0.520 8 newfstatat(3,"",0x00000055007ffcd0,0x1000) = 0
#14 0.520 8 mmap(NULL,213704,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500b47000
#14 0.520 8 mmap(0x0000005500b50000,148168,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FI
XED,3,0) = 0x0000005500b50000
#14 0.520 8 munmap(0x0000005500b47000,36864) = 0
#14 0.520 8 munmap(0x0000005500b75000,25288) = 0
#14 0.520 8 mprotect(0x0000005500b64000,61440,PROT_NONE) = 0
#14 0.520 8 mmap(0x0000005500b73000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIX
ED,3,0x13000) = 0x0000005500b73000
#14 0.520 8 close(3) = 0
#14 0.520 8 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libpthread.so.0",O_RDONLY|O_CLOEXEC) = 3
#14 0.520 8 read(3,0x7ffe20,832) = 832
#14 0.520 8 newfstatat(3,"",0x00000055007ffcb0,0x1000) = 0
#14 0.520 8 mmap(NULL,135200,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500b7c000
#14 0.520 8 mmap(0x0000005500b80000,69664,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIX
ED,3,0) = 0x0000005500b80000
#14 0.520 8 munmap(0x0000005500b7c000,16384) = 0
#14 0.520 8 munmap(0x0000005500b92000,45088) = 0
#14 0.520 8 mprotect(0x0000005500b81000,61440,PROT_NONE) = 0
#14 0.520 8 mmap(0x0000005500b90000,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIX
ED,3,0) = 0x0000005500b90000
#14 0.520 8 close(3) = 0
#14 0.520 8 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
#14 0.520 8 read(3,0x7ffe00,832) = 832
#14 0.520 8 newfstatat(3,"",0x00000055007ffc90,0x1000) = 0
#14 0.521 8 mmap(NULL,1805928,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500b9e000
#14 0.521 8 mmap(0x0000005500ba0000,1740392,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_F
IXED,3,0) = 0x0000005500ba0000
#14 0.521 8 munmap(0x0000005500b9e000,8192) = 0
#14 0.521 8 munmap(0x0000005500d49000,56936) = 0
#14 0.521 8 mprotect(0x0000005500d28000,61440,PROT_NONE) = 0
#14 0.521 8 mmap(0x0000005500d37000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FI
XED,3,0x187000) = 0x0000005500d37000
#14 0.521 8 mmap(0x0000005500d3d000,48744,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FI
XED,-1,0) = 0x0000005500d3d000
#14 0.521 8 close(3) = 0
#14 0.521 8 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500d
57000
#14 0.524 8 mmap(NULL,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x0000005500
d59000
#14 0.524 8 set_tid_address(365086213456,365080858688,365086213440,365080863120,365086213696,36
5086213632) = 8
#14 0.524 8 set_robust_list(365086213472,24,365086213472,1,0,365086213520) = -1 errno=38 (Funct
ion not implemented)
#14 0.524 8 Unknown syscall 293
#14 0.527 8 mprotect(0x0000005500d37000,16384,PROT_READ) = 0
#14 0.527 8 mprotect(0x0000005500b90000,4096,PROT_READ) = 0
#14 0.527 8 mprotect(0x0000005500b73000,4096,PROT_READ) = 0
#14 0.527 8 mprotect(0x0000005500b35000,4096,PROT_READ) = 0
#14 0.531 8 mprotect(0x0000005500a89000,45056,PROT_READ) = 0
#14 0.531 8 mprotect(0x0000005500860000,4096,PROT_READ) = 0
#14 0.533 8 mprotect(0x000000000523c000,16384,PROT_READ) = 0
#14 0.535 8 mprotect(0x000000550083b000,8192,PROT_READ) = 0
#14 0.535 8 prlimit64(0,3,0,365080611000,365085680408,320) = 0
#14 0.536 8 munmap(0x0000005500840000,7675) = 0
#14 0.538 8 getrandom(365086120240,8,1,365086213248,1,0) = 8
#14 0.539 8 brk(NULL) = 0x00000000052a8000
#14 0.539 8 brk(0x00000000052c9000) = 0x00000000052c9000
#14 0.543 8 futex(0x0000005500a977a4,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,2147483647,NULL,NULL,0) = 0
#14 0.573 8 brk(0x00000000052ea000) = 0x00000000052ea000
#14 0.582 8 clock_gettime(CLOCK_MONOTONIC,0x0000005500800c70) = 0 ({tv_sec = 13496,tv_nsec = 99
488400})
#14 0.582 8 gettimeofday(0x0000005500800c50,NULL) = 0 ({tv_sec = 1737120023,tv_usec = 146922},N
ULL)
#14 0.585 8 futex(0x000000000528ed68,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,2147483647,NULL,0x0000000000
000005,5) = 0
#14 0.606 8 brk(0x000000000530b000) = 0x000000000530b000
#14 0.622 8 rt_sigprocmask(SIG_SETMASK,0x0000005500800c30,0x0000005500800b10) = 0
#14 0.622 8 rt_sigaction(SIGILL,0x00000055008009b8,0x0000005500800a50) = 0
#14 0.623 8 rt_sigaction(SIGILL,0x00000055008009b8,NULL) = 0
#14 0.623 8 rt_sigprocmask(SIG_SETMASK,0x0000005500800b10,NULL) = 0
#14 0.640 8 getpid() = 8
#14 0.640 8 capget(0x55008008a0,0x55008008a8) = 0
#14 0.640 8 getuid() = 54902
#14 0.640 8 geteuid() = 54902
#14 0.640 8 getgid() = 54902
#14 0.640 8 getegid() = 54902
#14 0.642 8 fstat(0,0x000000000527e778) = 0
#14 0.642 8 fstat(1,0x000000000527e840) = 0
#14 0.642 8 fstat(2,0x000000000527e908) = 0
#14 0.642 8 rt_sigprocmask(SIG_SETMASK,0x0000005500800a80,NULL) = 0
#14 0.642 8 rt_sigaction(SIGHUP,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGINT,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGQUIT,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGILL,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(5,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGABRT,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(7,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGFPE,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGUSR1,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGSEGV,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGUSR2,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGPIPE,0x00000055008006a8,NULL) = 0
#14 0.642 8 rt_sigaction(SIGALRM,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGTERM,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(16,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGCHLD,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGCONT,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(20,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGTTIN,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(SIGTTOU,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(23,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(24,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(25,0x00000055008006a8,NULL) = 0
#14 0.642 8 rt_sigaction(26,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(27,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(28,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(29,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(30,NULL,0x0000005500800740) = 0
#14 0.642 8 rt_sigaction(31,NULL,0x0000005500800740) = 0
#14 0.643 8 fcntl(0,F_GETFL) = 0
#14 0.643 8 ioctl(0,TCGETS,0x0000005500800770) = -1 errno=25 (Not a tty)
#14 0.643 8 fstat(0,0x0000005500800830) = 0
#14 0.643 8 fcntl(1,F_GETFL) = 1
#14 0.643 8 ioctl(1,TCGETS,0x0000005500800770) = -1 errno=25 (Not a tty)
#14 0.643 8 fstat(1,0x0000005500800830) = 0
#14 0.643 8 fcntl(2,F_GETFL) = 1
#14 0.643 8 ioctl(2,TCGETS,0x0000005500800770) = -1 errno=25 (Not a tty)
#14 0.643 8 fstat(2,0x0000005500800830) = 0
#14 0.643 8 rt_sigaction(SIGINT,0x00000055008007f8,NULL) = 0
#14 0.643 8 rt_sigaction(SIGTERM,0x00000055008007f8,NULL) = 0
#14 0.643 8 prlimit64(0,7,0,365080611456,0,2147483648) = 0
#14 0.644 8 prlimit64(0,7,365080611456,0,0,2147483648) = 0
#14 0.644 8 clock_gettime(CLOCK_MONOTONIC,0x00000055008007e0) = 0 ({tv_sec = 13496,tv_nsec = 16
1099728})
#14 0.645 8 fcntl(0,F_SETFD,1) = 0
#14 0.645 8 fcntl(1,F_SETFD,1) = 0
#14 0.645 8 fcntl(2,F_SETFD,1) = 0
#14 0.645 8 fcntl(3,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(4,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(5,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(6,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(7,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(8,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(9,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(10,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(11,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(12,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(13,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(14,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(15,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.645 8 fcntl(16,F_SETFD,1) = -1 errno=9 (Bad file descriptor)
#14 0.651 8 getpid() = 8
#14 0.651 8 capget(0x55008006c0,0x55008006c8) = 0
#14 0.651 8 getuid() = 54902
#14 0.651 8 geteuid() = 54902
#14 0.651 8 getgid() = 54902
#14 0.651 8 getegid() = 54902
#14 0.651 8 getpid() = 8
#14 0.651 8 capget(0x55008006c0,0x55008006c8) = 0
#14 0.651 8 getuid() = 54902
#14 0.651 8 geteuid() = 54902
#14 0.651 8 getgid() = 54902
#14 0.651 8 getegid() = 54902
#14 0.651 8 getpid() = 8
#14 0.651 8 capget(0x55008006c0,0x55008006c8) = 0
#14 0.651 8 getuid() = 54902
#14 0.651 8 geteuid() = 54902
#14 0.651 8 getgid() = 54902
#14 0.651 8 getegid() = 54902
#14 0.652 8 getpid() = 8
#14 0.652 8 capget(0x55008006c0,0x55008006c8) = 0
#14 0.652 8 getuid() = 54902
#14 0.652 8 geteuid() = 54902
#14 0.652 8 getgid() = 54902
#14 0.652 8 getegid() = 54902
#14 0.652 8 getpid() = 8
#14 0.652 8 capget(0x55008007c0,0x55008007c8) = 0
#14 0.652 8 getuid() = 54902
#14 0.652 8 geteuid() = 54902
#14 0.652 8 getgid() = 54902
#14 0.652 8 getegid() = 54902
#14 0.658 8 getpid() = 8
#14 0.658 8 capget(0x55008007c0,0x55008007c8) = 0
#14 0.658 8 getuid() = 54902
#14 0.658 8 geteuid() = 54902
#14 0.658 8 getgid() = 54902
#14 0.658 8 getegid() = 54902
#14 0.658 8 futex(0x0000000005294960,FUTEX_PRIVATE_FLAG|FUTEX_WAKE,2147483647,NULL,NULL,0) = 0
#14 0.667 8 write(1,0x247a5d8,8) = 8
#14 0.667 v20.17.0
#14 0.667 8 write(1,0xd3c65b,1) = 1
#14 0.668 8 fstat(0,0x0000005500800a70) = 0
#14 0.668 8 fcntl(0,F_GETFL) = 0
#14 0.668 8 fstat(1,0x0000005500800a70) = 0
#14 0.668 8 fcntl(1,F_GETFL) = 1
#14 0.668 8 fstat(2,0x0000005500800a70) = 0
#14 0.668 8 fcntl(2,F_GETFL) = 1
#14 0.681 8 exit_group(0)
#14 0.686 1 wait4(-1,365114361732,0,0,365114633408,0) = 8
#14 0.686 --- SIGCHLD {si_signo=SIGCHLD, si_code=1, si_pid=8, si_uid=54902, si_status=0, si_uti
me=23, si_stime=2} ---
#14 0.686 1 rt_sigreturn(17,365072415712,1,0,365072404480,0) = -1 errno=513 (Successful exit fr
om sigreturn)
#14 0.686 1 wait4(-1,365114361732,1,0,365114633408,0) = -1 errno=10 (No child process)
#14 0.688 1 exit_group(0)
#14 DONE 0.8s

Systemd coredump:

  COREDUMP_UID=54902
    COREDUMP_GID=54902
    COREDUMP_SIGNAL_NAME=SIGSEGV
    COREDUMP_SIGNAL=11
    COREDUMP_RLIMIT=18446744073709551615
    COREDUMP_HOSTNAME=buildkitsandbox
    COREDUMP_COMM=node
    COREDUMP_EXE=/usr/bin/qemu-aarch64
    COREDUMP_SLICE=system.slice
    COREDUMP_PROC_LIMITS=Limit                     Soft Limit           Hard Limit           Units     
                         Max cpu time              unlimited            unlimited            seconds   
                         Max file size             unlimited            unlimited            bytes     
                         Max data size             unlimited            unlimited            bytes     
                         Max stack size            8388608              unlimited            bytes     
                         Max core file size        unlimited            unlimited            bytes     
                         Max resident set          unlimited            unlimited            bytes     
                         Max processes             unlimited            unlimited            processes 
                         Max open files            1024                 524288               files     
                         Max locked memory         8388608              8388608              bytes     
                         Max address space         unlimited            unlimited            bytes     
                         Max file locks            unlimited            unlimited            locks     
                         Max pending signals       61107                61107                signals   
                         Max msgqueue size         819200               819200               bytes     
                         Max nice priority         0                    0                    
                         Max realtime priority     0                    0                    
                         Max realtime timeout      unlimited            unlimited            us        
    COREDUMP_CWD=/home/git
    COREDUMP_ROOT=/
    MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1
    PRIORITY=2
    COREDUMP_PACKAGE_JSON={"elfType":"coredump","elfArchitecture":"AMD x86-64"}
    CODE_FILE=../systemd/src/coredump/coredump.c
    CODE_LINE=931
    CODE_FUNC=submit_coredump
    SYSLOG_IDENTIFIER=systemd-coredump
    _TRANSPORT=journal
    _UID=54902
    _GID=54902
    _COMM=systemd-coredum
    _EXE=/usr/lib/systemd/systemd-coredump
    _CMDLINE=/usr/lib/systemd/systemd-coredump
    _CAP_EFFECTIVE=0
    _SYSTEMD_SLICE=system-systemd\x2dcoredump.slice
    _MACHINE_ID=42c50f3f406e4854a6fc41b4d90bc6e2
    _HOSTNAME=archlinux
    _RUNTIME_SCOPE=system
    COREDUMP_PID=446520
    COREDUMP_TIMESTAMP=1737135759000000
    COREDUMP_CMDLINE=/usr/bin/qemu-aarch64 /opt/node-v20.17.0-linux-arm64/bin//node node --version
    COREDUMP_CGROUP=/system.slice/system.slice:docker:xyykonphxikhnir01ewmnw2wv
    COREDUMP_OPEN_FDS=0:pipe:[725533]
                      pos:      0
                      flags:    00
                      mnt_id:   16
                      ino:      725533
                      
                      1:pipe:[725534]
                      pos:      0
                      flags:    01
                      mnt_id:   16
                      ino:      725534
                      
                      2:pipe:[725535]
                      pos:      0
                      flags:    01
                      mnt_id:   16
                      ino:      725535
                      
                      3:/opt/node-v20.17.0-linux-arm64/bin/node
                      pos:      1024
                      flags:    0100040
                      mnt_id:   390
                      ino:      14204964
    COREDUMP_PROC_STATUS=Name:  node
                         Umask: 0022
                         State: S (sleeping)
                         Tgid:  446520
                         Ngid:  0
                         Pid:   446520
                         PPid:  446456
                         TracerPid:     0
                         Uid:   54902   54902   54902   54902
                         Gid:   54902   54902   54902   54902
                         FDSize:        64
                         Groups:        54902 
                         NStgid:        446520  8
                         NSpid: 446520  8
                         NSpgid:        446456  1
                         NSsid: 446456  1
                         Kthread:       0
                         VmPeak:          251248 kB
                         VmSize:          251248 kB
                         VmLck:        0 kB
                         VmPin:        0 kB
                         VmHWM:     3276 kB
                         VmRSS:     3276 kB
                         RssAnon:           1024 kB
                         RssFile:           2252 kB
                         RssShmem:             0 kB
                         VmData:          132800 kB
                         VmStk:      132 kB
                         VmExe:     3452 kB
                         VmLib:        8 kB
                         VmPTE:       44 kB
                         VmSwap:               0 kB
                         HugetlbPages:         0 kB
                         CoreDumping:   1
                         THP_enabled:   1
                         untag_mask:    0xffffffffffffffff
                         Threads:       2
                         SigQ:  0/61107
                         SigPnd:        0000000000000000
                         ShdPnd:        0000000000000000
                         SigBlk:        0000000000000000
                         SigIgn:        0000000000000000
                         SigCgt:        0000000000000000
                         CapInh:        0000000000000000
                         CapPrm:        0000000000000000
                         CapEff:        0000000000000000
                         CapBnd:        00000000a80425fb
                         CapAmb:        0000000000000000
                         NoNewPrivs:    0
                         Seccomp:       2
                         Seccomp_filters:       1
                         Speculation_Store_Bypass:      thread vulnerable
                         SpeculationIndirectBranch:     conditional enabled
                         Cpus_allowed:  ff
                         Cpus_allowed_list:     0-7
                         Mems_allowed:  00000001
                         Mems_allowed_list:     0
                         voluntary_ctxt_switches:       4
                         nonvoluntary_ctxt_switches:    0
                         x86_Thread_features:   
                         x86_Thread_features_locked:    
    COREDUMP_PROC_MAPS=00200000-0037b000 r--p 00000000 00:42 11177592                           /usr/bin/qemu-aarch64
                       0037b000-006da000 r-xp 0017a000 00:42 11177592                           /usr/bin/qemu-aarch64
                       006da000-0073d000 rw-p 004d8000 00:42 11177592                           /usr/bin/qemu-aarch64
                       0073d000-00749000 rw-p 0053a000 00:42 11177592                           /usr/bin/qemu-aarch64
                       00749000-0076b000 rw-p 00000000 00:00 0 
                       00b6b000-07a13000 ---p 00000000 00:00 0                                  [heap]
                       74aa4c800000-74aa547ff000 rwxp 00000000 00:00 0 
                       74aa547ff000-74aa54800000 ---p 00000000 00:00 0 
                       74aa54894000-74aa548aa000 rw-p 00000000 00:00 0 
                       74aa548ab000-74aa5498a000 rw-p 00000000 00:00 0 
                       74aa5498a000-74aa5498c000 ---p 00000000 00:00 0 
                       74aa5498c000-74aa549b7000 rw-p 00000000 00:00 0 
                       74aa549b7000-74aa549bb000 r--p 00000000 00:00 0                          [vvar]
                       74aa549bb000-74aa549bd000 r-xp 00000000 00:00 0                          [vdso]
                       7ffd75d41000-7ffd75d62000 rw-p 00000000 00:00 0                          [stack]
                       ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
    COREDUMP_PROC_CGROUP=0::/system.slice/system.slice:docker:xyykonphxikhnir01ewmnw2wv
    COREDUMP_PROC_MOUNTINFO=390 322 0:68 / / rw,relatime - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/HSTNAFFQXRES6QOB23>
                            392 390 0:79 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
                            393 390 0:80 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64
                            394 393 0:81 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
                            395 393 0:82 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k,inode64
                            396 393 0:77 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
                            397 390 0:83 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
                            398 390 259:2 /var/lib/docker/buildkit/executor/resolv.conf /etc/resolv.conf ro,nosuid,nodev,noexec,rela>
                            399 390 259:2 /var/lib/docker/buildkit/executor/hosts.5arcfkpmp4egnbjezv1mij6uz /etc/hosts ro,nosuid,nod>
                            400 397 0:28 /system.slice/system.slice:docker:xyykonphxikhnir01ewmnw2wv /sys/fs/cgroup ro,nosuid,nodev,>
                            323 392 0:79 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
                            324 392 0:79 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
                            325 392 0:79 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
                            354 392 0:79 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
                            355 392 0:79 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
                            356 392 0:84 / /proc/acpi ro,relatime - tmpfs tmpfs ro,inode64
                            357 392 0:85 / /proc/asound ro,relatime - tmpfs tmpfs ro,inode64
                            358 392 0:80 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64
                            362 392 0:80 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64
                            363 392 0:80 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64
                            364 397 0:86 / /sys/firmware ro,relatime - tmpfs tmpfs ro,inode64
                            365 397 0:87 / /sys/devices/virtual/powercap ro,relatime - tmpfs tmpfs ro,inode64
                            366 392 0:88 / /proc/scsi ro,relatime - tmpfs tmpfs ro,inode64
    COREDUMP_PROC_AUXV=[384B blob data]
    COREDUMP_CONTAINER_CMDLINE=/usr/bin/qemu-aarch64 /bin/sh /bin/sh -c $'node --version'
    COREDUMP_ENVIRON=HOME=/home/git
                     GIT_HOME_DIR=/home/git
                     QEMU_STRACE=1
                     PATH=/opt/node-v20.17.0-linux-arm64/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
                     CACHE_BUST=1
                     PWD=/home/git
    COREDUMP_FILENAME=/var/lib/systemd/coredump/core.node.54902.3aa96f3b05b243ae83d4b1841b85da42.446520.1737135759000000.zst
    MESSAGE=Process 446520 (node) of user 54902 dumped core.
            
            Module /usr/bin/qemu-aarch64 (deleted) without build-id.
            Stack trace of thread 8:
            #0  0x00000000006c91df n/a (/usr/bin/qemu-aarch64 (deleted) + 0x4c91df)
            ELF object binary architecture: AMD x86-64

Dockerfile:

FROM ubuntu:22.04

ENV GIT_HOME_DIR=/home/git

RUN apt-get update && apt-get -y install wget --no-install-recommends

RUN wget --no-check-certificate --quiet -O node.tar.gz \
    https://nodejs.org/download/release/v20.17.0/node-v20.17.0-linux-arm64.tar.gz

RUN tar -xf node.tar.gz --directory=/opt/

ENV PATH "/opt/node-v20.17.0-linux-arm64/bin/:$PATH"

RUN npm install -g npm@10.8.2

RUN apt-get update \
    && apt-get -y install build-essential make python3-minimal --no-install-recommends \
    && apt-get clean

RUN npm install -g yarn@^1.22

RUN groupadd --gid 54902 git \
    && useradd --comment "Jenkins User" --home-dir ${GIT_HOME_DIR} --uid 54902 --gid 54902 --create-home git
USER git
WORKDIR ${GIT_HOME_DIR}

ARG CACHE_BUST=1
RUN echo $CACHE_BUST

ENV QEMU_STRACE=1

RUN node --version
RUN npm --version
RUN yarn --version

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 12
  • Comments: 70 (16 by maintainers)

Commits related to this issue

Most upvoted comments

@ajbarber After some digging I’m pretty sure this issue relates to a kernel hardening. This also explains why various qemu versions are affected. More details can be found in this Debian bug: [1]. This bug first happened after [2] was applied (which later was reverted) and reverted again [3] after a fix for QEMU in Debian was available. Probably Ubuntu included just the kernel patch (revert-revert) but not the QEMU patch which then broke things again.

I’ve also started hitting segmentation faults today when building a multi-platform Docker image on GitHub Actions with a ubuntu-24.04 runner targeting linux/amd64 and linux/arm64 (using a common docker/setup-qemu-action@v3 and docker/setup-buildx-action@v3 setup) and found this issue report through https://github.com/docker/setup-qemu-action/issues/188 where a similar problem is described.

In my case, the segmentation faults occur when compiling the CPP application (that is deployed via the Docker image) during the build of the linux/arm64 image (so when QEMU is used):

#17 893.2 c++: internal compiler error: Segmentation fault signal terminated program cc1plus
#17 893.2 Please submit a full bug report, with preprocessed source (by using -freport-bug).
#17 893.2 See <file:///usr/share/doc/gcc-13/README.Bugs> for instructions.
#17 893.2 make[2]: *** [src/shared/CMakeFiles/shared.dir/build.make:510: src/shared/CMakeFiles/shared.dir/MailerService.cpp.o] Error 4
#17 893.3 make[2]: *** Deleting file 'src/shared/CMakeFiles/shared.dir/MailerService.cpp.o'
#17 893.3 make[2]: *** Waiting for unfinished jobs....
#17 912.1 make[1]: *** [CMakeFiles/Makefile2:444: src/shared/CMakeFiles/shared.dir/all] Error 2
#17 912.1 make: *** [Makefile:136: all] Error 2

The error occurs at random steps in the compilation process. I’ve had no issues building this Docker image hundreds of times over the past year until today; the last successful build with the same code revision before the segmentation faults started happening was at 00:00 GMT today (23.01.2025), so it looks like something must’ve changed/been updated since then (or I somehow got lucky before and didn’t hit the issue; that seems a bit unlikely though, considering I’ve had a segmentation fault in every single one of the six workflow runs I did today).

Edit: Using QEMU 8.1.5 (as suggested by @smoke) instead of 7.0.0 (which is currently still tagged as latest) seems to work for me:

- name: Set up QEMU
  uses: docker/setup-qemu-action@v3
  with:
    image: tonistiigi/binfmt:qemu-v8.1.5

Edit 2: Using QEMU 9.2.0 also works fine for me:

- name: Set up QEMU
  uses: docker/setup-qemu-action@v3
  with:
    image: tonistiigi/binfmt:qemu-v9.2.0

tonistiigi/binfmt:latest with QEMU 9.2.0 has been pushed: https://github.com/tonistiigi/binfmt/releases/tag/deploy%2Fv9.2.0-51

@ajbarber I suggest closing this issue. Most people used tonistiigi/binfmt:latest[^1] and ran into issues due to the ubuntu-24.04 GitHub Runner.

[^1]: tonistiigi/binfmt:latest currently (Feb 8, 2025) is (and has been since Aug 2, 2022) tonistiigi/binfmt:qemu-v7.0.0.

The reason for the real issue is described in https://github.com/tonistiigi/binfmt/issues/215#issuecomment-2613004741.

🤐

We can add the patches to https://github.com/tonistiigi/binfmt/tree/master/patches if you want to test master build.

@benz0li Ubuntu is not the only distro that affected by this. I have the same issue using Debian and Fedora with tonistiigi/binfmt:latest until I switched to qemu-user-static distro package.

Since old qemu versions are now broken due to upstream kernel hardening, I would recommend keeping this issue open until the latest docker tag gets updated.

I’m also encountering this. I’m using the github qemu action to build a variety of ruby versions for different architectures. Since the last 3 days, arm64-v8, ppc64le and s390x started to fail with pretty cryptic and random errors, both during configuring and building:

Some of the errors I encountered

  • make: *** [Makefile:333: libruby.so.3.5.0] Segmentation fault (core dumped)
  • make: *** [Makefile:462: miniinit.o] Error 139
  • make: *** [Makefile:462: miniinit.o] Error 1
  • configure: error: size_t is signed
  • configure: error: cannot compute sizeof (long long)
  • configure: error: cannot compute sizeof (dev_t)
  • configure: error: dup() must be supported
  • configure: error: something wrong with LDFLAGS=“”
  • configure: error: cannot compute suffix of object files: cannot compile
  • configure: error: cannot compute suffix of executables: cannot compile and link
  • configure: error: cannot run C compiled programs.
  • configure: error: ANSI C-conforming const and volatile are mandatory
  • gcc: internal compiler error: Segmentation fault signal terminated program cc1

I run my actions on ubuntu-24.04, and coincidentally there was a runner image update right when this started. I compared builds and with 20250105.1.0 they passed, while with 20250120.5.0 they failed. So, I find it likely that something that was updated in this changelog is responsible: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250120.5.

It updates buildx from 0.19.3 to 0.20.0, maybe it is their bug? I have no idea how to verify. The other software updates seem benign. On the other hand, this issue preceeds the buildx release by a few days so these may be two separate things.

Using tonistiigi/binfmt:qemu-v8.1.5 seems to solve it for me as well. Nevermind, I got another build failure. But it looks to be much more stable

Hitting similar issues when compiling php extensions while building a base image on top of php:8.3-fpm-bookworm

neither v7.0.0 nor 8.1.5 are working in this case.

I’ve got a similar thing with regular segfaults in gcc while building C or Go applications. It appears that when I create an image with a more recent version of qemu (8.1.5) which is the latest in the repo and use that to install the emulators, the segfault does not happen.

Is there a reason that the later versions (later than v7) have not been pushed to dockerhub as the latest version?

The fix seems to be included in version 9.2.2 @tonistiigi

9.2.0-51 didn’t fix the issue for me

7.0.0-28 works as expected.

@crazy-max same for me, 9.2 doesn’t solve the segmentation fault (reproducible in 4/4 runs with Ubuntu 22.4 and 24.4). Also had to revert to version 7.0.0-28. Please reopen the issue, as this error affects tens of thousands of people using the qemu GitHub action.

Thanks very much @fmoessbauer. Reading some of those materials you linked, the segfaulting in question of qemu depends not only on release versions, but also the configuration flags passed at build time.

We have maintainers of qemu saying clearly not to configure with --disable-pie or you will get crashes here. But it appears that is precisely what this repo is doing:

https://github.com/tonistiigi/binfmt/blob/85908cc7576a423dcf514eba5d05df8547c73513/scripts/configure_qemu.sh#L65

To confirm @fmoessbauer’s hypothesis, I also replicated the crash/no crash behaviour of qemu provided by this repo by turning off the aslr kernel parameter sysctl kernel.randomize_va_space=0 (off) and 2 (on). 2 leads to a crash as my original issue, 0 does not. If I put on the ubuntu 22.04 shipped qemu on the container instead, which is qemu-aarch64 version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.24), I can’t replicate the crash with aslr on or off. So in ubuntu’s packaging build scripts I’m guessing they have removed the --disable-pie, but I have not confirmed this.

I think qemu was forward patched in 8.1 to deal with things either way: https://gitlab.com/qemu-project/qemu/-/issues/1763#note_1508827541

So we need either to remove the line above in configure_qemu.sh or [edited for clarity] bump latest tag to qemu > 8.1.

@tonistiigi do you accept PRs?

I have stumbled on similar thing in different context and ecosystem https://github.com/docker/setup-qemu-action/issues/188

We have a Ruby app depends on quite a bunch of “native” extensions packages - that are packages having to be built through gcc

Targeting linux/amd64,linux/arm64

we have consistent Gem::Ext::BuildError: ERROR: Failed to build gem native extension. caused by gcc: internal compiler error: Segmentation fault signal terminated program cc1 but that will affect anything built by gcc or similar (nodejs packages, etc.).

Similar thing happened with a Go project needed some stuff built with gcc.

I have also managed to reproduce it locally with qemu v7.

What worked was updating to qemu v8, give it a try if it will help.

I’ve experienced it on 22.04 again, then i just switched to 24.04 in my GHA workflow and it started working again, so still a bit hit or miss, but depedent on the host machine either way it appears.

In both cases the latest image with QEMU 9.2.2 was pulled (and presumably used).

@tonistiigi @Earlopain

This worked for awhile, but started failing again with the exact same error… I was able to work around it as before with the following, but is it possible that 9.2.2-52 may have reintroduced the issue and a subsequent release resolved it again?

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          image: tonistiigi/binfmt:master

Do additional tests need to run before Docker Hub latest tag can be applied to the latest image?

Wow that did the trick. Thank you so much @bmhowe23.

We are still facing the segfault while building docker images for linux/arm64. For me, this happens even for different OS runners like Ubuntu 20.04 and 22.04.

I tried a few different combinations and they all faced the exact same segfault:

Looks like #244 is also able to reproduce the segfault on the latest 9.2.2.

@jaywonchung I think you likely need to put docker/setup-qemu-action before docker/setup-buildx-action in order for the binfmt version callout to take effect.

No, I made a PR to update qemu to 9.2.2 like suggested here. I was waiting for my builds to finish and with that qemu version everything is green for me as well.

Hopefully this works for others too and when the latest tag its switched over it’s finally fixed

The current theory is the kernel bump lead the runners into this issue https://gitlab.com/qemu-project/qemu/-/issues/1913 which was fixed only recently in version 9.2.2 which the container used with latest does not run yet.

He is talking about the docker/setup-qemu-action repository but it is not a solution.

Could be related to https://gitlab.com/qemu-project/qemu/-/issues/1913, fix was merged into master yesterday.

Unfortunately, just like @cagnulein reported, v9.2.0 also doesn’t help to fix the segmentation fault while trying to install libc-bin with Ubuntu 24.04.1 as base layer.

QEMU_STRACE
#12 104.8 2811 write(1,0x5f1fdd08,55)Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
#12 104.8  = 55
#12 104.8 2811 clock_gettime(CLOCK_REALTIME_COARSE,0x00007c1c5f2001e8) = 0 ({tv_sec = 1738952748,tv_nsec = 646200393})
#12 104.8 2811 write(7,0x5e906620,67) = 67
#12 104.8 2811 write(26,0x5e906350,31) = 31
#12 104.8 2811 clock_gettime(CLOCK_REALTIME_COARSE,0x00007c1c5f2001e8) = 0 ({tv_sec = 1738952748,tv_nsec = 646200393})
#12 104.8 2811 write(7,0x5e906620,74) = 74
#12 104.8 2811 write(26,0x5e906350,34) = 34
#12 104.8 2811 write(4,0x5e908920,1303) = 1303
#12 104.8 2811 ftruncate(4,1303) = 0
#12 104.8 2811 fsync(4) = 0
#12 104.8 2811 close(4) = 0
#12 104.8 2811 renameat(AT_FDCWD,"/var/lib/dpkg/updates/tmp.i",AT_FDCWD,"/var/lib/dpkg/updates/0204") = 0
#12 104.8 2811 openat(AT_FDCWD,"/var/lib/dpkg/updates/",O_RDONLY|O_DIRECTORY|O_NONBLOCK|O_CLOEXEC) = 4
#12 104.8 2811 newfstatat(4,"",0x00007c1c5f200308,0x1000) = 0
#12 104.8 2811 fsync(4) = 0
#12 104.8 2811 close(4) = 0
#12 104.8 2811 openat(AT_FDCWD,"/var/lib/dpkg/updates/tmp.i",O_WRONLY|O_CREAT|O_TRUNC,0666) = 4
#12 104.8 2811 fcntl(4,F_GETFD) = 0
#12 104.8 2811 fcntl(4,F_SETFD,1) = 0
#12 104.8 2811 newfstatat(4,"",0x00007c1c5f200208,0x1000) = 0
#12 104.8 2811 write(4,0x5e908920,4096) = 4096
#12 104.8 2811 write(4,0x5e908920,512) = 512
#12 104.8 2811 lseek(4,0,SEEK_SET) = 0
#12 104.8 2811 newfstatat(AT_FDCWD,"/var/lib/dpkg/info/libc-bin.postinst",0x00007c1c5f200248,0) = 0
#12 104.8 2811 clone(CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|0x11,child_stack=0x0000000000000000,parent_tidptr=0x0000000000000000,tls=0x0000000000000000,child_tidptr=0x00007c1c6789f0f0) = 3868
#12 104.8 2811 rt_sigaction(SIGQUIT,0x00007c1c5f1fff68,0x00007c1c5f200000) = 0
#12 104.8 2811 rt_sigaction(SIGINT,0x00007c1c5f1fff68,0x00007c1c5f200000) = 0
#12 104.8 2811 wait4(3868,0x7c1c5f200094,0,0) = 0
#12 104.8 3868 set_robust_list(0x7c1c6789f100,24) = -1 errno=38 (Function not implemented)
#12 104.8 3868 chdir("/") = 0
#12 104.8 3868 execve("/var/lib/dpkg/info/libc-bin.postinst",{"/var/lib/dpkg/info/libc-bin.postinst","triggered","ldconfig",NULL})3868 brk(NULL) = 0x000070be31d20000
#12 104.8 3868 uname(0x70be29a00418) = 0
#12 104.8 3868 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x000070be31c9f000
#12 104.8 3868 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,0) = -1 errno=2 (No such file or directory)
#12 104.8 3868 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
#12 104.8 3868 newfstatat(3,"",0x000070be299ff9c0,0x1000) = 0
#12 104.8 3868 mmap(NULL,9619,PROT_READ,MAP_PRIVATE,3,0) = 0x000070be31c9a000
#12 104.8 3868 close(3) = 0
#12 104.8 3868 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
#12 104.8 3868 read(3,0x299ffb70,832) = 832
#12 104.8 3868 newfstatat(3,"",0x000070be299ffa00,0x1000) = 0
#12 104.8 3868 mmap(NULL,1805928,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x000070be29a47000
#12 104.8 3868 mmap(0x000070be29a50000,1740392,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0) = 0x000070be29a50000
#12 104.8 3868 munmap(0x000070be29a47000,36864) = 0
#12 104.8 3868 munmap(0x000070be29bf9000,28264) = 0
#12 104.8 3868 mprotect(0x000070be29bd8000,61440,PROT_NONE) = 0
#12 104.8 3868 mmap(0x000070be29be7000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x187000) = 0x000070be29be7000
#12 104.8 3868 mmap(0x000070be29bed000,48744,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x000070be29bed000
#12 104.8 3868 close(3) = 0
#12 104.8 3868 set_tid_address(0x70be31c9ff50) = 3868
#12 104.8 3868 set_robust_list(0x70be31c9ff60,24) = -1 errno=38 (Function not implemented)
#12 104.8 3868 Unknown syscall 293
#12 104.8 3868 mprotect(0x000070be29be7000,16384,PROT_READ) = 0
#12 104.8 3868 mprotect(0x000070be31d1b000,8192,PROT_READ) = 0
#12 104.8 3868 mprotect(0x000070be31cea000,8192,PROT_READ) = 0
#12 104.8 3868 prlimit64(0,RLIMIT_STACK,NULL,0x000070be29a00588) = 0 ({rlim_cur=8388608,rlim_max=-1})
#12 104.8 3868 munmap(0x000070be31c9a000,9619) = 0
#12 104.8 3868 getuid() = 0
#12 104.8 3868 getgid() = 0
#12 104.8 3868 getpid() = 3868
#12 104.8 3868 rt_sigaction(SIGCHLD,0x000070be29a00538,NULL) = 0
#12 104.8 3868 geteuid() = 0
#12 104.8 3868 getrandom(0x70be29bf1930,8,1) = 8
#12 104.8 3868 brk(NULL) = 0x000070be31d20000
#12 104.8 3868 brk(0x000070be31d41000) = 0x000070be31d20000
#12 104.8 3868 mmap(NULL,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x000070be29100000
#12 104.8 3868 getppid() = 2811
#12 104.8 3868 newfstatat(AT_FDCWD,"/",0x000070be29a007b8,0) = 0
#12 104.8 3868 newfstatat(AT_FDCWD,".",0x000070be29a00838,0) = 0
#12 104.8 3868 openat(AT_FDCWD,"/var/lib/dpkg/info/libc-bin.postinst",O_RDONLY) = 3
#12 104.8 3868 fcntl(3,F_DUPFD,10) = 10
#12 104.8 3868 close(3) = 0
#12 104.8 3868 fcntl(10,F_SETFD,1) = 0
#12 104.8 3868 geteuid() = 0
#12 104.8 3868 getegid() = 0
#12 104.8 3868 rt_sigaction(SIGINT,NULL,0x000070be29a005a0) = 0
#12 104.8 3868 rt_sigaction(SIGINT,0x000070be29a00508,NULL) = 0
#12 104.8 3868 rt_sigaction(SIGQUIT,NULL,0x000070be29a005a0) = 0
#12 104.8 3868 rt_sigaction(SIGQUIT,0x000070be29a00508,NULL) = 0
#12 104.8 3868 rt_sigaction(SIGTERM,NULL,0x000070be29a005a0) = 0
#12 104.8 3868 rt_sigaction(SIGTERM,0x000070be29a00508,NULL) = 0
#12 104.8 3868 read(10,0x31d1dbd8,8192) = 1273
#12 104.8 3868 newfstatat(AT_FDCWD,"/usr/sbin/ldconfig",0x000070be29a002c8,0) = 0
#12 104.8 3868 rt_sigprocmask(SIG_SETMASK,[SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGIOT SIGBUS SIGFPE SIGKILL SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGSTKFLT SIGCHLD SIGCONT SIGSTOP SIGTSTP SIGTTIN SIGTTOU SIGURG SIGXCPU SIGXFSZ SIGVTALRM SIGPROF SIGWINCH SIGIO SIGPWR SIGSYS 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64],NULL,8) = 0
#12 104.8 3868 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x000070be29a00270,parent_tidptr=0x0000000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 3871
#12 104.8  = 0
#12 104.8 3868 rt_sigprocmask(SIG_SETMASK,[],0x000070be29a001e8,8) = 0 (oldset=[SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGIOT SIGBUS SIGFPE SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGSTKFLT SIGCHLD SIGCONT SIGTSTP SIGTTIN SIGTTOU SIGURG SIGXCPU SIGXFSZ SIGVTALRM SIGPROF SIGWINCH SIGIO SIGPWR SIGSYS 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60])
#12 104.8 3871 rt_sigprocmask(SIG_SETMASK,[],0x000070be29a001e8,8) = 0 (oldset=[SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGIOT SIGBUS SIGFPE SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGSTKFLT SIGCHLD SIGCONT SIGTSTP 3868 SIGTTINwait4(-1,0x70be29a00214,0,0) SIGTTOU SIGURG SIGXCPU SIGXFSZ SIGVTALRM SIGPROF SIGWINCH SIGIO SIGPWR SIGSYS 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60])
#12 104.8 3871 execve("/usr/sbin/ldconfig",{"ldconfig",NULL})3871 brk(NULL) = 0x00007a91264b0000
#12 104.8 3871 uname(0x7a911e200438) = 0
#12 104.8 3871 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007a912642f000
#12 104.8 3871 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,0) = -1 errno=2 (No such file or directory)
#12 104.8 3871 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
#12 104.8 3871 newfstatat(3,"",0x00007a911e1ff9e0,0x1000) = 0
#12 104.8 3871 mmap(NULL,9619,PROT_READ,MAP_PRIVATE,3,0) = 0x00007a912642a000
#12 104.8 3871 close(3) = 0
#12 104.8 3871 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
#12 104.8 3871 read(3,0x1e1ffb90,832) = 832
#12 104.8 3871 newfstatat(3,"",0x00007a911e1ffa20,0x1000) = 0
#12 104.8 3871 mmap(NULL,1805928,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007a911e247000
#12 104.8 3871 mmap(0x00007a911e250000,1740392,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0) = 0x00007a911e250000
#12 104.8 3871 munmap(0x00007a911e247000,36864) = 0
#12 104.8 3871 munmap(0x00007a911e3f9000,28264) = 0
#12 104.8 3871 mprotect(0x00007a911e3d8000,61440,PROT_NONE) = 0
#12 104.8 3871 mmap(0x00007a911e3e7000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x187000) = 0x00007a911e3e7000
#12 104.8 3871 mmap(0x00007a911e3ed000,48744,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x00007a911e3ed000
#12 104.8 3871 close(3) = 0
#12 104.8 3871 set_tid_address(0x7a912642ff50) = 3871
#12 104.8 3871 set_robust_list(0x7a912642ff60,24) = -1 errno=38 (Function not implemented)
#12 104.8 3871 Unknown syscall 293
#12 104.8 3871 mprotect(0x00007a911e3e7000,16384,PROT_READ) = 0
#12 104.8 3871 mprotect(0x00007a91264ab000,8192,PROT_READ) = 0
#12 104.8 3871 mprotect(0x00007a912647a000,8192,PROT_READ) = 0
#12 104.8 3871 prlimit64(0,RLIMIT_STACK,NULL,0x00007a911e2005a8) = 0 ({rlim_cur=8388608,rlim_max=-1})
#12 104.8 3871 munmap(0x00007a912642a000,9619) = 0
#12 104.8 3871 getuid() = 0
#12 104.8 3871 getgid() = 0
#12 104.8 3871 getpid() = 3871
#12 104.8 3871 rt_sigaction(SIGCHLD,0x00007a911e200558,NULL) = 0
#12 104.8 3871 geteuid() = 0
#12 104.8 3871 getrandom(0x7a911e3f1930,8,1) = 8
#12 104.8 3871 brk(NULL) = 0x00007a91264b0000
#12 104.8 3871 brk(0x00007a91264d1000) = 0x00007a91264b0000
#12 104.8 3871 mmap(NULL,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007a911d900000
#12 104.8 3871 getppid() = 3868
#12 104.8 3871 newfstatat(AT_FDCWD,"/",0x00007a911e2007d8,0) = 0
#12 104.8 3871 newfstatat(AT_FDCWD,".",0x00007a911e200858,0) = 0
#12 104.8 3871 openat(AT_FDCWD,"/usr/sbin/ldconfig",O_RDONLY) = 3
#12 104.8 3871 fcntl(3,F_DUPFD,10) = 10
#12 104.8 3871 close(3) = 0
#12 104.8 3871 fcntl(10,F_SETFD,1) = 0
#12 104.8 3871 geteuid() = 0
#12 104.8 3871 getegid() = 0
#12 104.8 3871 rt_sigaction(SIGINT,NULL,0x00007a911e2005c0) = 0
#12 104.8 3871 rt_sigaction(SIGINT,0x00007a911e200528,NULL) = 0
#12 104.8 3871 rt_sigaction(SIGQUIT,NULL,0x00007a911e2005c0) = 0
#12 104.8 3871 rt_sigaction(SIGQUIT,0x00007a911e200528,NULL) = 0
#12 104.8 3871 rt_sigaction(SIGTERM,NULL,0x00007a911e2005c0) = 0
#12 104.8 3871 rt_sigaction(SIGTERM,0x00007a911e200528,NULL) = 0
#12 104.8 3871 read(10,0x264adbd8,8192) = 387
#12 104.8 3871 execve("/sbin/ldconfig.real",{"/sbin/ldconfig.real",NULL})3871 brk(NULL) = 0x00007e715e9fa000
#12 104.8 3871 brk(0x00007e715e9fab98) = 0x00007e715e9fa000
#12 104.8 --- SIGSEGV {si_signo=SIGSEGV, si_code=1, si_addr=0x0000000000000040} ---
#12 104.8 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#12 104.8  = 
#12 104.8 --- SIGCHLD {si_signo=SIGCHLD, si_code=3, si_pid=3871, si_uid=0, si_status=11, si_utime=2, si_stime=1} ---
#12 104.8 3868 rt_sigreturn(0x11) = -1 errno=513 (Successful exit from sigreturn)
#12 104.8 3868 write(2,0x29a00298,33)Segmentation fault (core dumped)
#12 104.8  = 33
#12 104.8 3868 wait4(-1,0x70be29a00214,1,0) = -1 errno=10 (No child process)
#12 104.8 3868 rt_sigprocmask(SIG_SETMASK,[SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGIOT SIGBUS SIGFPE SIGKILL SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGSTKFLT SIGCHLD SIGCONT SIGSTOP SIGTSTP SIGTTIN SIGTTOU SIGURG SIGXCPU SIGXFSZ SIGVTALRM SIGPROF SIGWINCH SIGIO SIGPWR SIGSYS 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64],NULL,8) = 0
#12 104.8 3868 clone(CLONE_VM|CLONE_VFORK|0x11,child_stack=0x000070be29a00270,parent_tidptr=0x0000000000000000,tls=0x0000000000000000,child_tidptr=0xfffffffe7fffffff) = 3875
#12 104.8 3868 rt_sigprocmask(SIG_SETMASK,[],0x000070be29a001e8,8) = 0 (oldset=[SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGIOT SIGBUS SIGFPE SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGSTKFLT SIGCHLD SIGCONT SIGTSTP SIGTTIN SIGTTOU SIGURG SIGXCPU SIGXFSZ SIGVTALRM SIGPROF SIGWINCH SIGIO SIGPWR SIGSYS 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 =  49 050
#12 104.8  51 52 53 54 55 56 57 58 59 60])
#12 104.8 3868 wait4(-1,0x70be29a00214,0,0)3875 rt_sigprocmask(SIG_SETMASK,[],0x000070be29a001e8,8) = 0 (oldset=[SIGHUP SIGINT SIGQUIT SIGILL SIGTRAP SIGIOT SIGBUS SIGFPE SIGUSR1 SIGSEGV SIGUSR2 SIGPIPE SIGALRM SIGTERM SIGSTKFLT SIGCHLD SIGCONT SIGTSTP SIGTTIN SIGTTOU SIGURG SIGXCPU SIGXFSZ SIGVTALRM SIGPROF SIGWINCH SIGIO SIGPWR SIGSYS 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60])
#12 104.8 3875 execve("/usr/sbin/ldconfig",{"ldconfig","--verbose",NULL})3875 brk(NULL) = 0x00007ac0f07f0000
#12 104.8 3875 uname(0x7ac0f0600428) = 0
#12 104.8 3875 mmap(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007ac0f8813000
#12 104.8 3875 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,0) = -1 errno=2 (No such file or directory)
#12 104.8 3875 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3
#12 104.8 3875 newfstatat(3,"",0x00007ac0f05ff9d0,0x1000) = 0
#12 104.8 3875 mmap(NULL,9619,PROT_READ,MAP_PRIVATE,3,0) = 0x00007ac0f880e000
#12 104.8 3875 close(3) = 0
#12 104.8 3875 openat(AT_FDCWD,"/lib/aarch64-linux-gnu/libc.so.6",O_RDONLY|O_CLOEXEC) = 3
#12 104.8 3875 read(3,0xf05ffb80,832) = 832
#12 104.8 3875 newfstatat(3,"",0x00007ac0f05ffa10,0x1000) = 0
#12 104.8 3875 mmap(NULL,1805928,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007ac0efc47000
#12 104.8 3875 mmap(0x00007ac0efc50000,1740392,PROT_EXEC|PROT_READ,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0) = 0x00007ac0efc50000
#12 104.8 3875 munmap(0x00007ac0efc47000,36864) = 0
#12 104.8 3875 munmap(0x00007ac0efdf9000,28264) = 0
#12 104.8 3875 mprotect(0x00007ac0efdd8000,61440,PROT_NONE) = 0
#12 104.8 3875 mmap(0x00007ac0efde7000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_DENYWRITE|MAP_FIXED,3,0x187000) = 0x00007ac0efde7000
#12 104.8 3875 mmap(0x00007ac0efded000,48744,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x00007ac0efded000
#12 104.8 3875 close(3) = 0
#12 104.8 3875 set_tid_address(0x7ac0f8813f50) = 3875
#12 104.8 3875 set_robust_list(0x7ac0f8813f60,24) = -1 errno=38 (Function not implemented)
#12 104.8 3875 Unknown syscall 293
#12 104.8 3875 mprotect(0x00007ac0efde7000,16384,PROT_READ) = 0
#12 104.8 3875 mprotect(0x00007ac0f07eb000,8192,PROT_READ) = 0
#12 104.8 3875 mprotect(0x00007ac0f07ba000,8192,PROT_READ) = 0
#12 104.8 3875 prlimit64(0,RLIMIT_STACK,NULL,0x00007ac0f0600598) = 0 ({rlim_cur=8388608,rlim_max=-1})
#12 104.8 3875 munmap(0x00007ac0f880e000,9619) = 0
#12 104.8 3875 getuid() = 0
#12 104.8 3875 getgid() = 0
#12 104.8 3875 getpid() = 3875
#12 104.8 3875 rt_sigaction(SIGCHLD,0x00007ac0f0600548,NULL) = 0
#12 104.8 3875 geteuid() = 0
#12 104.8 3875 getrandom(0x7ac0efdf1930,8,1) = 8
#12 104.8 3875 brk(NULL) = 0x00007ac0f07f0000
#12 104.8 3875 brk(0x00007ac0f0811000) = 0x00007ac0f07f0000
#12 104.8 3875 mmap(NULL,1048576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007ac0f067c000
#12 104.8 3875 getppid() = 3868
#12 104.8 3875 newfstatat(AT_FDCWD,"/",0x00007ac0f06007c8,0) = 0
#12 104.8 3875 newfstatat(AT_FDCWD,".",0x00007ac0f0600848,0) = 0
#12 104.8 3875 openat(AT_FDCWD,"/usr/sbin/ldconfig",O_RDONLY) = 3
#12 104.8 3875 fcntl(3,F_DUPFD,10) = 10
#12 104.8 3875 close(3) = 0
#12 104.8 3875 fcntl(10,F_SETFD,1) = 0
#12 104.8 3875 geteuid() = 0
#12 104.8 3875 getegid() = 0
#12 104.8 3875 rt_sigaction(SIGINT,NULL,0x00007ac0f06005b0) = 0
#12 104.8 3875 rt_sigaction(SIGINT,0x00007ac0f0600518,NULL) = 0
#12 104.8 3875 rt_sigaction(SIGQUIT,NULL,0x00007ac0f06005b0) = 0
#12 104.8 3875 rt_sigaction(SIGQUIT,0x00007ac0f0600518,NULL) = 0
#12 104.8 3875 rt_sigaction(SIGTERM,NULL,0x00007ac0f06005b0) = 0
#12 104.8 3875 rt_sigaction(SIGTERM,0x00007ac0f0600518,NULL) = 0
#12 104.8 3875 read(10,0xf07edbd8,8192) = 387
#12 104.8 3875 execve("/sbin/ldconfig.real",{"/sbin/ldconfig.real","--verbose",NULL})3875 brk(NULL) = 0x0000752dbccfa000
#12 104.8 3875 brk(0x0000752dbccfab98) = 0x0000752dbccfa000
#12 104.8 --- SIGSEGV {si_signo=SIGSEGV, si_code=1, si_addr=0x0000000000000040} ---
#12 104.8 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#12 104.8  = 
#12 104.8 --- SIGCHLD {si_signo=SIGCHLD, si_code=3, si_pid=3875, si_uid=0, si_status=11, si_utime=2, si_stime=1} ---
#12 104.8 3868 rt_sigreturn(0x11) = -1 errno=513 (Successful exit from sigreturn)
#12 104.8 3868 write(2,0x29a00298,33)Segmentation fault (core dumped)
#12 104.8  = 33
#12 104.8 3868 wait4(-1,0x70be29a00214,1,0) = -1 errno=10 (No child process)
#12 104.8 3868 exit_group(139)
#12 104.8  = 
#12 104.8 [2811](https://cicd.skyway.porsche.com/oso/ort/-/jobs/39287804#L2811) write(2,0x5f1fda48,150)dpkg: error processing package libc-bin (--configure):
#12 104.8  installed libc-bin package post-installation script subprocess returned error exit status 139
#12 104.8  = 150

A similiar problem has already been discussed in this issue: https://github.com/docker/buildx/issues/1170, but changing the whole build command to

docker context create gitlab
docker buildx create --name multiarch-builder --use gitlab
docker run --rm --privileged tonistiigi/binfmt:qemu-v9.2.0 --install all
docker run --rm --privileged multiarch/qemu-user-static --reset --persistent yes --credential yes
docker buildx build ...

didn’t help either. The build then starts again to segfault at arbritrary points in time:

#60 76.09 gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DECP_NISTZ256_ASM -DKECCAK1600_ASM -DOPENSSL_BN_ASM_MONT -DOPENSSL_CPUID_OBJ -DPOLY1305_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DVPAES_ASM -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/opt/rbenv/versions/3.3.4/openssl/ssl\"" -DENGINESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/engines-3\"" -DMODULESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG  -MMD -MF crypto/async/libcrypto-lib-async_err.d.tmp -MT crypto/async/libcrypto-lib-async_err.o -c -o crypto/async/libcrypto-lib-async_err.o crypto/async/async_err.c
#60 76.19 gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DECP_NISTZ256_ASM -DKECCAK1600_ASM -DOPENSSL_BN_ASM_MONT -DOPENSSL_CPUID_OBJ -DPOLY1305_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DVPAES_ASM -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/opt/rbenv/versions/3.3.4/openssl/ssl\"" -DENGINESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/engines-3\"" -DMODULESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG  -MMD -MF crypto/async/libcrypto-lib-async_wait.d.tmp -MT crypto/async/libcrypto-lib-async_wait.o -c -o crypto/async/libcrypto-lib-async_wait.o crypto/async/async_wait.c
#60 76.23 gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DECP_NISTZ256_ASM -DKECCAK1600_ASM -DOPENSSL_BN_ASM_MONT -DOPENSSL_CPUID_OBJ -DPOLY1305_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DVPAES_ASM -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/opt/rbenv/versions/3.3.4/openssl/ssl\"" -DENGINESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/engines-3\"" -DMODULESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG  -MMD -MF crypto/bf/libcrypto-lib-bf_cfb64.d.tmp -MT crypto/bf/libcrypto-lib-bf_cfb64.o -c -o crypto/bf/libcrypto-lib-bf_cfb64.o crypto/bf/bf_cfb64.c
#60 76.26 gcc: internal compiler error: Segmentation fault signal terminated program cc1
#60 76.26 gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DECP_NISTZ256_ASM -DKECCAK1600_ASM -DOPENSSL_BN_ASM_MONT -DOPENSSL_CPUID_OBJ -DPOLY1305_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DVPAES_ASM -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/opt/rbenv/versions/3.3.4/openssl/ssl\"" -DENGINESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/engines-3\"" -DMODULESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG  -MMD -MF crypto/bf/libcrypto-lib-bf_ecb.d.tmp -MT crypto/bf/libcrypto-lib-bf_ecb.o -c -o crypto/bf/libcrypto-lib-bf_ecb.o crypto/bf/bf_ecb.c
#60 76.26 Please submit a full bug report,
#60 76.26 with preprocessed source if appropriate.
#60 76.26 See <file:///usr/share/doc/gcc-11/README.Bugs> for instructions.
#60 76.26 gcc  -I. -Iinclude -Iproviders/common/include -Iproviders/implementations/include  -DECP_NISTZ256_ASM -DKECCAK1600_ASM -DOPENSSL_BN_ASM_MONT -DOPENSSL_CPUID_OBJ -DPOLY1305_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DVPAES_ASM -fPIC -pthread -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSLDIR="\"/opt/rbenv/versions/3.3.4/openssl/ssl\"" -DENGINESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/engines-3\"" -DMODULESDIR="\"/opt/rbenv/versions/3.3.4/openssl/lib/ossl-modules\"" -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG  -MMD -MF crypto/bf/libcrypto-lib-bf_enc.d.tmp -MT crypto/bf/libcrypto-lib-bf_enc.o -c -o crypto/bf/libcrypto-lib-bf_enc.o crypto/bf/bf_enc.c
#60 76.27 make[1]: *** [Makefile:13373: crypto/async/libcrypto-lib-async.o] Error 4
#60 76.27 make[1]: *** Waiting for unfinished jobs....
#60 77.88 make[1]: Leaving directory '/tmp/ruby-build.20250207185021.44.aZZcoI/openssl-3.0.15'

I tried a lot of different combinations, but the build either crashes when using gcc or when installing libc-bin. To be honest, I’m very confused and wondering, if tonistiigi/binfmt:qemu-v9.2.0 and multiarch/qemu-user-static should even be used together.

You can try with tonistiigi/binfmt:qemu-v9.2.0, it’s not tagged to latest yet but will look into it next week.

Just tried it and tonistiigi/binfmt:qemu-v9.2.0 works without issues for my setup, like tonistiigi/binfmt:qemu-v8.1.5 does as well.

  • name: Setup QEMU uses: docker/setup-qemu-action@v3 with: image: tonistiigi/binfmt:master

Same issue unfortunately

Setting up qtbase5-private-dev:arm64 (5.15.2+dfsg-9+deb11u1) ... Setting up libqt5texttospeech5-dev:arm64 (5.15.2-2) ... Setting up qtpositioning5-dev:arm64 (5.15.2+dfsg-2) ... Setting up qtlocation5-dev:arm64 (5.15.2+dfsg-2) ... Processing triggers for libc-bin (2.31-13+deb11u11) ... qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault (core dumped) qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault (core dumped) dpkg: error processing package libc-bin (--configure): installed libc-bin package post-installation script subprocess returned error exit status 139

tonistiigi/binfmt:master is now on top of QEMU 9.2.0 if you want to test:

$ docker run --privileged --rm tonistiigi/binfmt:master --version
Unable to find image 'tonistiigi/binfmt:master' locally
master: Pulling from tonistiigi/binfmt
3282d2a020fe: Pull complete
2ae55cc90af8: Pull complete
Digest: sha256:cafe00b517cf8482cbf3e8ac35c2a477e1a0934fa2786cee953fa14eb03012db
Status: Downloaded newer image for tonistiigi/binfmt:master
binfmt/14ebeca qemu/v9.2.0 go/1.23.5

Reverting from what version?

@ajbarber From tonistiigi/binfmt:latest which I wrongly assumed it points to the latest build, but no it’s a 2 years old image…

@cagnulein in my case the GH runner image is ubuntu-24.04 v20250120.5.0, the buildx multi-arch CGO build used to fail constantly, but with tonistiigi/binfmt:qemu-v8.1.5-43 works fine, 6 builds and so far no more seg fault.

Same issue with Ubuntu 22-04 unfortunately

docker/setup-qemu-action uses tonistiigi/binfmt:latest by default. ℹ This can be customised.

I have also managed to reproduce it locally with qemu v7.

When you say locally, do you mean outside of a container, just on the CPU?

I’ve just tried to reproduce with simply qemu-aarch64 v7.2.15, and an ARM64 node binary on my AMD64 metal (no docker) and I can’t replicate the segfault.

It could be a qemu-aarch64 user binary issue, or an issue with a c/c++ runtime library on the target container. One way to eliminate the container Linux C/C++ libraries being the cause is to update the Linux version running in docker and see if the problem persists.

(Edited, realised I’m running into this on both ubuntu 20.04 and 22.04)

Thanks @smoke. This repo only supports up to v7 though right? In the meantime I will try and get a stack, I can see it has been removed in the core dump logout above.