podman: Running ubi9 on Apple silicon breaks on unsupported x86_64-v2 instruction set

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description I’m trying to build and run an AMD64 ubi9 image on Apple silicon. I have installed qemu-user-binfmt in the Fedora CoreOS VM, but when I start a podman build with a Containerfile that starts with FROM registry.access.redhat.com/ubi9/ubi-minimal:latest, the build fails with

Fatal glibc error: CPU does not support x86-64-v2
  1. On Apple silicon, brew install podman, set up a VM

  2. Try and run a build with a Containerfile that has a FROM registry.access.redhat.com/ubi9/ubi-minimal:latest and with --arch amd64

  3. See error

Describe the results you received:

✗ podman build --arch amd64 -f Containerfile -t testing:0.1 .
STEP 1/7: FROM registry.access.redhat.com/ubi9/ubi-minimal:latest
Trying to pull registry.access.redhat.com/ubi9/ubi-minimal:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:743b2fff6512b752ffd624ae8b45a951ca8378f3d84a912061cdaf508db1a29d
Copying blob sha256:961425e673a750fb2d17875c2a4bf5a319c8681e4fdf81a5a6e2d819cef95e35
Copying config sha256:4a8128b051b8bfafcc6bd540db324a5244aff1ef7bd9f6dda7ecd10680c4fbbe
Writing manifest to image destination
Storing signatures
STEP 2/7: RUN microdnf install -y python3 python3-pip
Fatal glibc error: CPU does not support x86-64-v2
Error: error building at STEP "RUN microdnf install -y python3 python3-pip": error while running runtime: exit status 127

Describe the results you expected: Successful build

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.5
Built:        Wed Aug 10 22:46:05 2022
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Thu Aug 11 16:43:11 2022
OS/Arch:      linux/arm64

Output of podman info:

host:
  arch: arm64
  buildahVersion: 1.27.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 92.23
    systemPercent: 6.17
    userPercent: 1.6
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "36"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 502
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.18.18-200.fc36.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1673756672
  memTotal: 2051903488
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/502/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.aarch64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 0h 2m 56.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106825756672
  graphRootUsed: 2977878016
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/502/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1660228991
  BuiltTime: Thu Aug 11 16:43:11 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/arm64
  Version: 4.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

brew info podman
==> podman: stable 4.2.0 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.2.0 (178 files, 48MB) *
  Poured from bottle on 2022-08-11 at 13:56:31
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go-md2man ✘, go@1.18 ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
Bash completion has been installed to:
  /opt/homebrew/etc/bash_completion.d
==> Analytics
install: 21,840 (30 days), 57,845 (90 days), 195,310 (365 days)
install-on-request: 21,411 (30 days), 57,072 (90 days), 194,478 (365 days)
build-error: 37 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes (that is, the 4.2.0 version in brew)

Additional environment details (AWS, VirtualBox, physical, etc.): Apple M1

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 37 (29 by maintainers)

Commits related to this issue

Most upvoted comments

It’s a limitation in qemu-user-static-x86 binfmt support. rhel9 expects a CPU supporting the x86_64-v2 instruction set, but qemu-user-static binfmt support is configured to run qemu-x86_64-static directly (see /usr/lib/binfmt.d/qemu-x86_64-static.conf

ssh’ing into a linux VM running on the M1 and running ld-linux-x86_64.so.2 --help returns:

$ qemu-x86_64-static ./ld-linux-x86-64.so.2 --help |grep x86-64
Usage: ./ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2
  x86-64-v4
  x86-64-v3
  x86-64-v2

It’s possible to use --cpu as an argument to qemu-x86_64-static to get something better:

$ qemu-x86_64-static --cpu Haswell-v2  ./ld-linux-x86-64.so.2 --help |grep x86-64
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.01H:ECX.avx [bit 28]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.01H:ECX.f16c [bit 29]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.07H:EBX.avx2 [bit 5]
qemu-x86_64-static: warning: TCG doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
Usage: ./ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
This program interpreter self-identifies as: /lib64/ld-linux-x86-64.so.2
  x86-64-v4
  x86-64-v3
  x86-64-v2 (supported, searched)

(the important bit is the last line)

I tried to configure binfmt to use an intermediate bash script or go program to start qemu-x86_64-static with the correct arguments, this worked when trying to start an x86_64 binary directly from the command line inside the VM, but when I then try to use podman run, this fails:

$ podman run  --arch x86_64 -it --rm registry.access.redhat.com/ubi9-minimal /usr/bin/ls
2022/08/24 15:55:48 no such file or directory

(with an intermediate go binary). With a shell script, this gives an error message about too many symlinks. I’m not quite sure what’s wrong/what’s confusing podman.

I have written a quick and dirty program to test the feature level of an x86 CPU here (it will also list which features of a feature level is supported and which are missing): https://github.com/CKingX/x86-feature-test/releases/tag/v1.0.0 This may be of interest to this issue.

this means they are using a modified/patched binary. It might be good to pick this up with @dustymabe and Fedora too.

The original request was handled here, but this was only the break up of the packages: coreos/fedora-coreos-tracker#1088

We just install the qemu-user-static-x86 package. Any issues would need to be taken up with qemu (link to bugzilla can be found here: https://src.fedoraproject.org/rpms/qemu )

Responding to question in Twitter whether this problem also occurs with Docker instead of Podman:

docker run --platform linux/amd64 -it --rm registry.access.redhat.com/ubi9-minimal cat /proc/cpuinfo
Unable to find image 'registry.access.redhat.com/ubi9-minimal:latest' locally
latest: Pulling from ubi9-minimal
961425e673a7: Pull complete
743b2fff6512: Pull complete
Digest: sha256:4ba4d3e2da3a7edb8a8e79fd38e95702f445aa4d5a66a1b1667260b5f1405acc
Status: Downloaded newer image for registry.access.redhat.com/ubi9-minimal:latest
processor	: 0
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x00
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

processor	: 1
BogoMIPS	: 48.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint
CPU implementer	: 0x00
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x000
CPU revision	: 0

The issue is about how to get this correctly configured to allow binfmt_misc and podman to correctly run the binaries with the changed -cpu setting.