podman: Running a pod as rootless user with systemd: "setting rlimit type 7: operation not permitted"

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

According to #5605: I try to run a pod with systemd as a rootless user and one redis container inside, I get the same error and some additional output if a extend the logging:

Jun 18 12:16:58 hostsystem systemd[1]: Started Podman pod-redis-pod.service.
Jun 18 12:16:58 hostsystem systemd[1]: Starting Podman container-redis.service...
Jun 18 12:16:59 hostsystem podman[24348]: Error: unable to start container "redis": container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:353: setting rlimits for ready process caused \\\"error setting rlimit type 7: operation not permitted\\\"\"": OCI runtime permission denied error
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Control process exited, code=exited, status=125/n/a
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Failed with result 'exit-code'.
Jun 18 12:16:59 hostsystem systemd[1]: Failed to start Podman container-redis.service.
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Service RestartSec=100ms expired, scheduling restart.
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Scheduled restart job, restart counter is at 5.
Jun 18 12:16:59 hostsystem systemd[1]: Stopped Podman container-redis.service.
Jun 18 12:16:59 hostsystem systemd[1]: Stopping Podman pod-redis-pod.service...

Steps to reproduce the issue:

  1. Create pod and container:
podman pod create -n redis-pod -p 6379
podman create --name=redis -d --pod=redis-pod localhost/redis:6.0.4
podman generate systemd -f -n redis-pod
  1. Modify unit files like this:
[Unit]
Description=Podman pod-redis-pod.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
Requires=container-redis.service
Before=container-redis.service

[Service]
User=rootlessuser
Group=rootlessuser
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman --log-level debug --syslog true start 2cf7fac7e0d9-infra
ExecStop=/usr/bin/podman stop -t 10 2cf7fac7e0d9-infra
PIDFile=/run/user/2662/containers/overlay-containers/d16d7cfc632aaac4c2a6e118a21ff8c8031af9fccc9cd1ae42280d7dd66e9df0/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

[Unit]
Description=Podman container-redis.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RefuseManualStart=yes
RefuseManualStop=yes
BindsTo=pod-redis-pod.service
After=pod-redis-pod.service

[Service]
User=rootlessuser
Group=rootlessuser
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start redis
ExecStop=/usr/bin/podman stop -t 10 redis
PIDFile=/run/user/2662/containers/overlay-containers/2e0e9d201129d04071b7cedb8e6d828c7877684979b9a66e8f4280749538c9c3/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target
  1. Copy unit files to /etc/systemd/system/ and enable them.

Describe the results you received:

Here is a debug log which is created by systemd running the above unit. It is one try of some where systemd tries to start the pod.

tl;dr

Jun 18 12:16:56 hostsystem systemd[1]: Starting Podman container-redis.service...
Jun 18 12:16:56 hostsystem podman[23798]: Error: unable to start container "redis": container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:353: setting rlimits for ready process caused \\\"error setting rlimit type 7: operation not permitted\\\"\"": OCI runtime permission denied error
Jun 18 12:16:56 hostsystem systemd[1]: container-redis.service: Control process exited, code=exited, status=125/n/a
Jun 18 12:16:56 hostsystem systemd[1]: container-redis.service: Failed with result 'exit-code'.
Jun 18 12:16:56 hostsystem systemd[1]: Failed to start Podman container-redis.service.
Jun 18 12:16:57 hostsystem systemd[1]: container-redis.service: Service RestartSec=100ms expired, scheduling restart.
Jun 18 12:16:57 hostsystem systemd[1]: container-redis.service: Scheduled restart job, restart counter is at 3.
Jun 18 12:16:57 hostsystem systemd[1]: Stopped Podman container-redis.service.
Jun 18 12:16:57 hostsystem systemd[1]: Stopping Podman pod-redis-pod.service...
Jun 18 12:16:57 hostsystem podman[23900]: d16d7cfc632aaac4c2a6e118a21ff8c8031af9fccc9cd1ae42280d7dd66e9df0
Jun 18 12:16:57 hostsystem systemd[1]: pod-redis-pod.service: Succeeded.
Jun 18 12:16:57 hostsystem systemd[1]: Stopped Podman pod-redis-pod.service.
Jun 18 12:16:57 hostsystem systemd[1]: pod-redis-pod.service: Found left-over process 23169 (fuse-overlayfs) in control group while starting unit. Ignoring.
Jun 18 12:16:57 hostsystem systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 18 12:16:57 hostsystem systemd[1]: pod-redis-pod.service: Found left-over process 23722 (slirp4netns) in control group while starting unit. Ignoring.
Jun 18 12:16:57 hostsystem systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 18 12:16:57 hostsystem systemd[1]: Starting Podman pod-redis-pod.service...

After systemd stopped restarting, this was logged by the pod:

$ podman logs 2cf7fac7e0d9-infra
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated

Describe the results you expected:

A running pod with a redis container inside.

Additional information you deem important (e.g. issue happens only occasionally):

This issue happens everytime with 1.9.3.

Output of podman version:

Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.11.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.11.6
  podmanVersion: 1.9.3
host:
  arch: amd64
  buildahVersion: 1.14.9
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.16, commit: '
  cpus: 24
  distribution:
    distribution: debian
    version: "10"
  eventLogger: file
  hostname: hostsystem
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 1083040
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 1083040
      size: 65536
  kernel: 4.19.0-9-amd64
  memFree: 58359988224
  memTotal: 66994249728
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0~rc6+dfsg1
      commit: 1.0.0~rc6+dfsg1-3
      spec: 1.0.1
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.0
      commit: unknown
      libslirp: 4.2.0
  swapFree: 7999582208
  swapTotal: 7999582208
  uptime: 48h 29m 16.66s (Approximately 2.00 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/rootlessuser/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/rootlessuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/2662/containers
  volumePath: /home/rootlessuser/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 1.9.3~1 amd64 [installed]
podman/unknown 1.9.3~1 arm64
podman/unknown 1.9.3~1 armhf
podman/unknown 1.9.3~1 ppc64el

Additional environment details (AWS, VirtualBox, physical, etc.):

I do not have this problem if I use podman 1.9.1 with the same setting.

In Release v1.9.2 you fixed a bug: “Fixed a bug where rootless Podman was setting resource limits on cgroups v2 systems that were not using systemd-managed cgroups (and thus did not support resource limits), resulting in containers failing to start” Maybe this could be the issue, that podman now tries to use cgroupv2 which is not enabled if it is running wiht systemd? My idea now is to enable cgroupv2 and use crun.

I also tried this workaround (with some changes) and it worked for me. But it is not the way I want to use podman.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 36 (18 by maintainers)

Most upvoted comments

Rootful should change to use cgroupfs for --cgroup-manager. You can permanently change this in containers.conf.
There is a fix in the works for the rootless issues on podman trying to set pids-limit.

https://github.com/containers/libpod/pull/6842

+1
rhatdan on Jul 4, 2020

I get the same error Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:378: setting rlimits for ready process caused \\\"error setting rlimit type 7: invalid argument\\\"\"": OCI runtime error

with a simple podman run --rm -it hello-world as non-root.

podman run --rm -it --ulimit host hello-world works.

+1
the-antz on Jun 24, 2020

Thanks, @giuseppe!

@rei-ber , could try the mentioned workarounds/solutions in #6389. #6389 (comment) could be one.

Well, I add this to ~/.config/containers/containers.conf:

default_ulimits = [
 "nofile=4098:4098",
]

and it worked more or less. First I got this error in journalctl while I enabled debuglog and syslog in start command of podman:

Jun 22 14:28:45 hostsystem podman[10157]: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

After podman system reset and creating a new containers.conf with the settings above in users’ home, it worked. I did not made any changes under /etc/security/limits.conf So my default hard limit for this user is:

$ ulimit -Hn
1048576

But I recognize another behavior. I have another container in the same pod with redis inside. The other container is created with --ulimit host. After starting the pod, the infra container and redis container try to start, but they fail due to the failed to write to /proc/self/oom_score_adj: Permission denied error. If I remove --ulimit host from the other container specification and have nofile ulimit set via containers.conf it works. Why does it not work with --ulimit host? I would expect that it takes the config from containers.conf?!

+1
rei-ber on Jun 22, 2020
Read more comments on GitHub
← podman: RUN fails in rootless podman build when cgroups is v1
podman: Running "podman images -a" is very slow →