toolbox: Cannot enter container – crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

Describe the bug A clear and concise description of what the bug is. If possible, re-run the command(s) with --log-level debug and put the output here.

Steps how to reproduce the behaviour

  1. Uhm… just created a new Fedora 38 container (toolbox create)
  2. Deleted old Fedora 36 and 37 containers (toolbox rm fedora-toolbox-36 etc.)
  3. Rebooted (and applied some rpm-ostree updates by that)
  4. And then I could not enter the container with toolbox enter anymore.

Expected behaviour Can enter and not suddently, after rebooting, fail to enter my container… 😦

Actual behaviour

$ toolbox enter                          
Error: failed to start container fedora-toolbox-38

Output of toolbox --version (v0.0.90+)

$ toolbox --version
toolbox version 0.0.99.4

Toolbox package info (rpm -q toolbox)

$ rpm -q toolbox
toolbox-0.0.99.4-1.fc38.x86_64

Output of podman version e.g.,

$ podman version
Client:       Podman Engine
Version:      4.5.1
API Version:  4.5.1
Go Version:   go1.20.4
Built:        Fri May 26 19:58:48 2023
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)

podman-4.5.1-1.fc38.x86_64

Info about your OS Fedora Silverblue 38

$ rpm-ostree status -v
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● fedora:fedora/38/x86_64/silverblue (index: 0)
                  Version: 38.20230612.0 (2023-06-12T00:45:27Z)
[…]

Additional context

I mean, you had one job, and this failed… I mean, no offense, I am just sad…

$ toolbox enter -vv             
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Looking for sub-GID and sub-UID ranges for user rugk 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/rugk/.config/toolbox 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called version.PersistentPreRunE(podman --log-level debug version --format json) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/rugk/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/rugk/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/rugk/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/rugk/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Called version.PersistentPostRunE(podman --log-level debug version --format json) 
DEBU[0000] Shutting down engines                        
DEBU Current Podman version is 4.5.1              
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 4.5.1                  
DEBU Migration not needed: Podman version 4.5.1 is unchanged 
DEBU Setting up configuration                     
DEBU Setting up configuration: file /var/home/rugk/.config/containers/toolbox.conf not found 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-38'               
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution (CLI): ''                       
DEBU Image (CLI): ''                              
DEBU Release (CLI): ''                            
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-38'               
DEBU Image: 'fedora-toolbox:38'                   
DEBU Release: '38'                                
DEBU Checking if container fedora-toolbox-38 exists 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exists.PersistentPreRunE(podman --log-level debug container exists fedora-toolbox-38) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/rugk/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/rugk/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/rugk/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/rugk/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Called exists.PersistentPostRunE(podman --log-level debug container exists fedora-toolbox-38) 
DEBU[0000] Shutting down engines                        
DEBU Inspecting mounts of container fedora-toolbox-38 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-38) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/rugk/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/rugk/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/rugk/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/rugk/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 49             
DEBU[0000] Looking up image "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" in local containers storage 
DEBU[0000] Trying "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" ... 
DEBU[0000] parsed reference into "[overlay@/var/home/rugk/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" 
DEBU[0000] Found image "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" as "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" in local containers storage 
DEBU[0000] Found image "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" as "997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f" in local containers storage ([overlay@/var/home/rugk/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]@997b52ccbf8544c42851a181e80bcd0f081eff8a879256b67d273a7e07f31f6f) 
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-38) 
DEBU[0000] Shutting down engines                        
DEBU Starting container fedora-toolbox-38         
Error: failed to start container fedora-toolbox-38

When I run podman manually:

$ podman start fedora-toolbox-38
Error: unable to start container "f8accc0c103a4fc741b8592de53010f8630d502d97a0050c367e2401cac1501f": crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

$ toolbox list -i
IMAGE ID      IMAGE NAME                                    CREATED
215122d241c2  registry.fedoraproject.org/fedora-toolbox:36  13 months ago
90d416a5811e  registry.fedoraproject.org/fedora-toolbox:37  6 months ago
997b52ccbf85  registry.fedoraproject.org/fedora-toolbox:38  2 months ago

$ podman start --attach fedora-toolbox-38 
Error: unable to start container f8accc0c103a4fc741b8592de53010f8630d502d97a0050c367e2401cac1501f: crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

Related issues

Search only turned up https://github.com/containers/toolbox/issues/1297, but I use no VirtualBox. Alos tried https://github.com/containers/podman/issues/14284#issuecomment-1130146224 (again altghough I donÄt use VirtualBox), and it kinda works, but is still throws erros (and is no solution of course):

$ podman run --rm -it --privileged --group-add keep-groups fedora-toolbox:38 
WARN[0000] Error validating CNI config file /var/home/rugk/.config/cni/net.d/87-podman.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]] 
[root@d3693d8a2176 /]#

edit: forget the command line above, it starts a fresh container based on the …:38 image, so it’s not my existing container instance…

I have no idea what config files these are…

$ cat /var/home/rugk/.config/cni/net.d/87-podman.conflist                  
{
  "cniVersion": "0.4.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni-podman0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [{ "dst": "0.0.0.0/0" }],
        "ranges": [
          [
            {
              "subnet": "10.88.0.0/16",
              "gateway": "10.88.0.1"
            }
          ]
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "tuning"
    }
  ]
}

Also found this file, which seems to be created at reboot time(?):

$ cat $XDG_CONFIG_HOME/toolbox/podman-system-migrate 
4.5.1
$ ls -la $XDG_CONFIG_HOME/toolbox/podman-system-migrate
-rw-r--r--. 1 rugk rugk 6 12. Jun 22:20 /var/home/rugk/.config/toolbox/podman-system-migrate

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 2
  • Comments: 25 (4 by maintainers)

Most upvoted comments

based on this: https://github.com/containers/podman/issues/19634#issuecomment-1680734973

I created a script to migrate old/broken toolboxes to new ones. This error will keep persisting on older containers regardless of having an up to date podman version because the limits were previously set in the container spec by older versions of podman. Newer versions don’t have this behaviour but the only way to fix this is to recreate the container with a newer podman version, just upgrading won’t solve it.

Anyway, this is a script that will export, reimport and recreate new toolboxes based on the ones you have currently on your system:

podman ps -a --format json \
    | jq -r '.[] | select(.Labels."com.github.containers.toolbox").Names[0]' \
    | xargs -tI{} \
        bash -c 'podman export {} -o {}.tar.gz && podman import {}.tar.gz {}-image && toolbox create --image localhost/{}-image {}-new'

the manual steps are as follows:

podman export -o $container_name.tar.gz $container_name
podman import $container_name.tar.gz $container_name-image
toolbox create --image $container_name-image $container_name-new

The new containers will be named with a suffix -new to avoid conflicts with older toolboxes, the script will also produce tarballs on PWD corresponding to each toolbox on the system

+7
akdev1l on Aug 21, 2023

The new containers will be named with a suffix -new to avoid conflicts with older toolboxes

A hint so people do not need to waste time on searching: You can then simply rename the container like this:

$ podman rename ${containerName}-new ${containerName}
$ podman rename fedora-toolbox-38-new fedora-toolbox-38
+2
rugk on Aug 21, 2023

soru is still getting this issue, even with podman v4.6.1

+1
Sorunome on Aug 19, 2023

I have “recovered” from this issue, by using this. My current ulimit is 200000 which is less than the ~256000 I get a boot time. I artificially reduce it to 100000 (ulimit -u 100000). I then follow the steps to recreate the containers, resulting in them capturing the 100000 setting. When I start the recreated containers, my limit will be greater than 100000, so podman start reducing it to 100000 works.

Eventually someone will fix this and it will make it into my rpm-ostree and hopefully I will not have to save/restore/recreate the toolboxen again.

+1
permezel on Jul 6, 2023

I am hitting the same problem

+1
RishabhSaini on Jun 29, 2023

I’m hitting this same problem with the exact same crun-1.8.5-1.fc38

+1
mcatanzaro on Jun 13, 2023
Read more comments on GitHub
← storage: `podman images` slow with `--uidmap`
toolbox: Container doesn't start because of missing /media possibly due to failed RPM transaction →