podman: Unable to start toolbox containers in Fedora Silverblue 31

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description Unable to start fedora-toolbox-31 (or any other toolbox container) in Fedora Silverblue 31.

Steps to reproduce the issue:

  1. toolbox create to create a new toolbox on Fedora Silverblue 31

  2. toolbox enter to enter the toolbox. This generates an error: toolbox: failed to start container fedora-toolbox-31

  3. podman --log-level debug start fedora-toolbox-31 to debug the issue generates a warning (see description)`

Describe the results you received:

The following output is presented when debuging with:

podman --log-level debug start fedora-toolbox-31

INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /var/home/returntrip/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/returntrip/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/returntrip/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/returntrip/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] overlay: mount_data=lowerdir=/var/home/returntrip/.local/share/containers/storage/overlay/l/HDZBVTBU4P5YJODD2W5VAHROK6:/var/home/returntrip/.local/share/containers/storage/overlay/l/UNHS2OY3XUUJNRAEMMDBKQJA23,upperdir=/var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/diff,workdir=/var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/work,context="system_u:object_r:container_file_t:s0:c687,c748" 
DEBU[0000] mounted container "2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c" at "/var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/merged" 
DEBU[0000] Created root filesystem for container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c at /var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c to user.slice:libpod:2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c at /var/home/returntrip/.local/share/containers/storage/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -s -c 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c -u 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c -r /usr/bin/crun -b /var/home/returntrip/.local/share/containers/storage/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata -p /run/user/1000/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/pidfile -l k8s-file:/var/home/returntrip/.local/share/containers/storage/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/returntrip/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: write unix @: sendmsg: broken pipe 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Cleaning up container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c" 
ERRO[0000] unable to start container "fedora-toolbox-31": error reading container (probably exited) json message: EOF 

Describe the results you expected:

User should be able to access toolbox without errors

Additional information you deem important (e.g. issue happens only occasionally): Discussed with Rishi and he suggested to file an issue here

Output of podman version:

podman version 1.5.1-dev

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13rc1
  podman version: 1.5.1-dev
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: 118fcdfca36d706f766bad2663b11bd2c41bf2e7'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 7146295296
  MemTotal: 16778080256
  OCIRuntime:
    package: crun-0.8-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun 0.8
      spec: 1.0.0
      +SYSTEMD +SELINUX +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 7985950720
  SwapTotal: 7985950720
  arch: amd64
  cpus: 16
  eventlogger: journald
  hostname: rauros.figura.io
  kernel: 5.3.0-0.rc6.git0.1.fc31.x86_64
  os: linux
  rootless: true
  uptime: 1h 55m 38.6s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/returntrip/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/returntrip/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000
  VolumePath: /var/home/returntrip/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.): Phisical machine. Rebased FSB30 to FSB31

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 36 (25 by maintainers)

Most upvoted comments

I too am seeing this issue, and after applying overrides for podman and crun, and unsetting NOTIFY_SOCKET, containers still fail to start.

Versions: podman 1.6.0-2.gitca5ff03.fc31 crun 0.9.1-1.fc31

conmon version 2.0.0
commit: ae68c1c29156934457e2649990c241a5e0b0251b

Logs:

INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/evan/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/evan/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/1000                     
DEBU[0000] Using static dir /home/evan/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/evan/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/home/evan/.local/share/containers/storage/overlay/l/A6ZQYUBMXOV5BFSH5OFYXP6OXM:/home/evan/.local/share/containers/storage/overlay/l/YQZ3CD5TJRRAEAT4AM5RCW2DQH,upperdir=/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/diff,workdir=/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/work,context="system_u:object_r:container_file_t:s0:c423,c497" 
DEBU[0000] mounted container "c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640" at "/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged" 
DEBU[0000] Created root filesystem for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 at /var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 to user.slice:libpod:c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 at /home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 -u c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 -r /usr/bin/crun -b /home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata -p /tmp/1000/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/pidfile -l k8s-file:/home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /tmp/1000/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/evan/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640]"
DEBU[0000] disabling SD notify                          
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640" 
ERRO[0000] unable to start container "fedora-toolbox-31": statfs '/var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged/etc/profile.d/toolbox.sh': No such file or directory: OCI runtime command not found error 
Sep 19 00:29:40 rob systemd[1403]: libpod-0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172.scope: Succeeded.
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <error>: Failed to create container: exit status 1
Sep 19 00:29:40 rob systemd[1403]: Started libcrun container.
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: terminal_ctrl_fd: 12
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: ctl fifo path: /home/evan/.local/share/containers/storage/overlay-containers/0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172/userdata/ctl
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172/attach}
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172/attach
Sep 19 00:29:40 rob conmon[10284]: conmon 0b9fc319c19a5c7778c7 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 19 00:29:40 rob systemd[1403]: Started /usr/bin/podman --log-level debug start fedora-toolbox-31.