podman: unable to run rootless container: error with setrlimit RLIMIT_NPROC

/kind bug

Description

I’m unable to run a rootless container, podman returns the following error:

$ podman run --rm golang:1.14-alpine go version
Error: setrlimit `RLIMIT_NPROC`: Invalid argument: OCI runtime error

I’m on Fedora 32. I’m sure running a rootless container worked with Fedora 31 but I’m not sure if the problem appeared as soon as I migrated to Fedora 32 or if it appeared later since I don’t use podman that often.

Steps to reproduce the issue:

podman run --rm golang:1.14-alpine go version (or any image really)

Describe the results you received:

Error: setrlimit `RLIMIT_NPROC`: Invalid argument: OCI runtime error

Describe the results you expected:

I expect the container to run correctly.

Additional information you deem important (e.g. issue happens only occasionally):

I tried:

uninstalling podman, libpod, conmon, container-selinux, containers-common
removing the directories $HOME/.local/share/containers and $HOME/.config.containers
reinstalling podman

but the error remains.

Output of podman version:

Version:            1.9.2
RemoteAPI Version:  1
Go Version:         go1.14.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.14.2
  podmanVersion: 1.9.2
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.16-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.16, commit: 1044176f7dd177c100779d1c63931d6022e419bd'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: thor
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.14-300.fc32.x86_64+debug
  memFree: 3684970496
  memTotal: 16685846528
  ociRuntime:
    name: crun
    package: crun-0.13-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.0.0-1.fc32.x86_64
    version: |-
      slirp4netns version 1.0.0
      commit: a3be729152a33e692cd28b52f664defbf2e7810a
      libslirp: 4.2.0
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 1h 11m 1.23s (Approximately 0.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/vincent/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.0.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/vincent/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 8
  runRoot: /run/user/1000/containers
  volumePath: /home/vincent/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.9.2-1.fc32.x86_64

It’s a desktop PC.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 68 (32 by maintainers)

Most upvoted comments

I just faced the same issue after updating kernel from 6.2.x to 6.3.1. Recreating the containers worked.

+14

kriansa on May 4, 2023

Using fedora kinoite, i’ve noticed changes in max user processes (ulimit -u -H) sometimes increase sometimes a decrease, when it decreases below the value it was when i created the container it basically causes this issue but i do not know why it keeps on changing

sandorex on May 29, 2023

for anyone out there still having this issue, you have to recreate the affected containers

Sorry what does it mean? Can I recreate the containers keeping all theirs data?

bam80 on Aug 25, 2020

for anyone out there still having this issue, you have to recreate the affected containers

Sorry what does it mean? Can I recreate the containers keeping all theirs data?

FWIW, I managed to backup my containers in F32, then rebase on F33 and restore the backups using this tutorial: https://fedoramagazine.org/backup-and-restore-toolboxes-with-podman/

philn on Oct 3, 2020

You don’t have to re-create the containers: https://unix.stackexchange.com/questions/127777/how-to-configure-the-process-open-file-limit-of-a-user

travier on Jul 25, 2023

I just encountered this issue today on Fedora Silverblue after an update. Here’s the weird bit:

$ podman inspect my-distrobox --format '{{ .HostConfig.Ulimits }}'
[{RLIMIT_NOFILE 524288 524288} {RLIMIT_NPROC 127724 127724}]
$ ulimit -u
127723

It’s off-by-one (127724 vs 127723). A bit obnoxious, considering that I have to go recreate my toolboxes 😦

J3RN on Jul 25, 2023

I don’t use docker/podman that heavily, my normal workflow is to just start what I need in my project with

podman-compose up -d

If your case is similar, a quick workaround for this issue could be this:

podman-compose up -d --force-recreate

eiglow on May 22, 2023

@llunved please share your limits.conf thanks The container in question had NPROC set to 62509, while the system default was 62508:

# podman inspect --format '{{ printf "%+v" .HostConfig.Ulimits }}' fedora-toolbox-32 [{Name:RLIMIT_NOFILE Soft:524288 Hard:524288} {Name:RLIMIT_NPROC Soft:62509 Hard:62509}]

So I added the following line to limits.conf (my user is in the group wheel - but you could use another one or the user):

@wheel hard nproc 62509

llunved on Oct 4, 2020

I have a similar situation. A container that was working a few days/weeks ago no longer works:

$ podman start artifactory
Error: unable to start container "e6976c2c158683a0db0bdec79095791b66f8ad697558f90a81eccb06cec41212": crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

I checked the container’s limit:

$ podman inspect --format '{{ printf "%+v" .HostConfig.Ulimits }}' artifactory    
[{Name:RLIMIT_NOFILE Soft:32000 Hard:32000} {Name:RLIMIT_NPROC Soft:62796 Hard:62796}

These were just one above the current (default) hard limit on the system:

$ ulimit -u
62795

I have no custom ulimits set on the system. I added the following to /etc/security/limits.d/11-podman.conf for now to fix this:

myuser - nproc 62796

Not sure what changed or when. System is Arch Linux with Linux 5.17.2-arch2-1. After logging in again to set the new limits I was able to launch the container.

alanorth on Apr 15, 2022

I got same error message when I try to restart the running pod. It started happening just today, was working totally fine few days before. No changes in limits.conf

Fodoj on Sep 24, 2020

just ran into the same issue create a new file /etc/security/limits.d/100-podman.conf with the following content

<username> hard nproc <value>

where value is the max value gather from RLIMIT_NPROC using podman inspect my-distrobox --format '{{ .HostConfig.Ulimits }}' then logout and log back in and just reboot

profit! no need to recreate the containers

bubuntux on Aug 15, 2023

I got my existing toolbox container working by setting the nproc limit for my user in /etc/security/limits.conf to the value reported for the existing container with podman inspect --format ‘{{ printf “%+v” .HostConfig.Ulimits }}’

llunved on Sep 7, 2020

Ok with some of my older containers, I am seeing these fields being set.

$ podman inspect charming_ride --format '{{ .HostConfig.Ulimits }}'
[{RLIMIT_NOFILE 300 300} {RLIMIT_NPROC 50 50}]

But newer containers I am creating with podman 2.0. Does not create the Ulimits.

$ podman create -ti alpine sh
fc7abf0fddfa30e1c375e44f0c70f180ff63be8a19f816cbbcb46d292a76f750
$ podman inspect -l --format '{{ .HostConfig.Ulimits }}'
[]

rhatdan on Jun 22, 2020

FWIW I started getting this error on Arch Linux also once Linux 5.7 hit the core repositories a few days ago. I’d previously been running rootless containers on this system just fine for months.

I am also using cgroups v1:

$ podman info | grep cgroup
  cgroupVersion: v1

Edit: Oops, I just downgraded my kernel to 5.6.15 on Arch and I still can’t run any containers. Something else must have broken it. For reference:

$ podman version
Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.14.3
Git Commit:         5d44534fff6877b1cb15b760242279ae6293154c
Built:              Mon May 25 22:25:50 2020
OS/Arch:            linux/amd64

I’ve definitely run containers in the past week, so it must have broken sometime around then.

alanorth on Jun 15, 2020