podman: Fedora IoT 34 ARM - Cannot Start Pods

/kind bug

Description

After upgrading to Fedora 34, it hasn’t been possible to start any pods (root or rootless) on my ARM system. Single containers work fine. On my ARM64 system, the issue does not exist.

Steps to reproduce the issue:

1. Upgrade to Fedora 34

2. Start Pods (through systemd or otherwise)

3. Notice that only single containers have started

Describe the results you received: 2021-05-03T16:42:53.000943269Z: container b7a... is not running ERRO[0007] error starting some container dependencies
ERRO[0007] “/usr/bin/crun start b7a... failed: exit status 1”

Describe the results you expected: The pod cluster to start normally

Additional information you deem important (e.g. issue happens only occasionally): Only seems to happen on Fedora 34 IoT with ARM. Output of podman version:

Version: 3.1.2 API Version: 3.1.2 Go Version: go1.16 Built: Thu Apr 22 13:13:16 2021 OS/Arch: linux/arm

Output of podman info --debug:

host: arch: arm buildahVersion: 1.20.1 cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.0.27-2.fc34.armv7hl path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: ’ cpus: 4 distribution: distribution: fedora version: “34” eventLogger: journald hostname: pi idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.11.16-300.fc34.armv7hl linkmode: dynamic memFree: 180670464 memTotal: 1007710208 ociRuntime: name: crun package: crun-0.19.1-2.fc34.armv7hl path: /usr/bin/crun version: |- crun version 0.19.1 commit: 1535fedf0b83fb898d449f9680000f729ba719f5 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: exists: true path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true selinuxEnabled: true slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.9-1.fc34.armv7hl version: |- slirp4netns version 1.1.8+dev commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.0 swapFree: 0 swapTotal: 0 uptime: 51m 44.73s registries: search:

• registry.fedoraproject.org
• registry.access.redhat.com
• docker.io
• quay.io store: configFile: /var/home/pi/.config/containers/storage.conf containerStore: number: 16 paused: 0 running: 6 stopped: 10 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.5.0-1.fc34.armv7hl Version: |- fusermount3 version: 3.10.2 fuse-overlayfs: version 1.5 FUSE library version 3.10.2 using FUSE kernel interface version 7.31 graphRoot: /var/home/pi/.local/share/containers/storage graphStatus: Backing Filesystem: extfs Native Overlay Diff: “false” Supports d_type: “true” Using metacopy: “false” imageStore: number: 58 runRoot: /run/user/1000/containers volumePath: /var/home/pi/.local/share/containers/storage/volumes version: APIVersion: 3.1.2 Built: 1619097196 BuiltTime: Thu Apr 22 13:13:16 2021 GitCommit: “” GoVersion: go1.16 OsArch: linux/arm Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.1.2-1.fc34.armv7hl

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Checked the guide, did not compile Podman from source

Additional environment details (AWS, VirtualBox, physical, etc.): Physical Raspberry Pi 3B+ box running Fedora IoT and is almost exclusively managed by Podman and Cockpit. There are no other software installed on it.