podman: failed to write to /proc/self/oom_score_adj: Permission denied

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running rootless podman I’m getting failed to write to /proc/self/oom_score_adj: Permission denied error, but container runs after it.

Steps to reproduce the issue:

podman run -it --rm anycontainer

Describe the results you received: The container runs, but after I get an error message in the terminal

Describe the results you expected: The container should start without error message.

Additional information you deem important (e.g. issue happens only occasionally): Error message: failed to write to /proc/self/oom_score_adj: Permission denied Output of podman version:

podman version 1.2.0

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.11.6
  podman version: 1.2.0
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2.0-1.1.x86_64
    path: /usr/lib/podman/bin/conmon
    version: "failed to write to /proc/self/oom_score_adj: Permission denied, conmon
      version 1.14.0\ncommit: "
  Distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20190423"
  MemFree: 2402082816
  MemTotal: 11992973312
  OCIRuntime:
    package: runc-1.0.0~rc6-3.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  SwapFree: 2145845248
  SwapTotal: 2147479552
  arch: amd64
  cpus: 4
  hostname: kraken
  kernel: 5.0.8-1-default
  os: linux
  rootless: true
  uptime: 4h 43m 30.3s (Approximately 0.17 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ConfigFile: /home/dario/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/dario/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 20
  RunRoot: /tmp/1000
  VolumePath: /home/dario/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.): physical

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 56 (14 by maintainers)

Most upvoted comments

I ran into the same issue on Fedora 38. Per a comment in #19930 I downgraded crun sudo dnf downgrade crun and that seems to have worked for now.

+10

kdubois on Sep 25, 2023

Update: This solved it for me:

edit /usr/lib/systemd/system/user@.service and remove the following line.

OOMScoreAdjust=100 https://github.com/systemd/systemd/commit/ce7de0ba8ef355c5119e7dffc46e075b257dc90d#diff-e712f4f510835ff8be07a088402d790305356f9284023c7a286056ccf38147e3R28

Reboot

Source: https://bbs.archlinux.org/viewtopic.php?pid=2012166#p2012166

bosd on Sep 17, 2023

I have updated from Fedora Silverblue 38 to 39 and one of the first things I did was to check whether podman is still working. Nope! All rootless containers fail with

crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied

Since /usr is write protected in Fedora Silverblue I have copied user@.service:

cp /usr/lib/systemd/system/user@.service /etc/systemd/system

I can confirm that after a reboot rootless container start again.

podman version 4.6.2 crun version 1.9

fansari on Oct 1, 2023

My laptop updated cri-o this morning, 1.13.5 to 1.13.7, and the oom_score_adj warning is gone.

edsantiago on May 2, 2019

@stemid appreciate that comment, I’ve not attempted this yet on my systems (which are centos9stream gitlab runners)… wondering if it would be better to consider the override mechanism within systemd.

Yeah that is actually what I ended up doing, my comment was a bit premature. I ended up using the equivalent of systemctl edit user@.service but since I deploy on CoreOS with Ignition that means I created the directory /etc/systemd/system/user@.service.d with the file override.conf and just two lines;

[Service]
OOMScoreAdjust=

stemid on Oct 14, 2023

I recently updated to podman 4.6.2 on fedora 38 and noticed this issue as well.

bruno-fs on Sep 19, 2023

I have updated to Podman 4.6.2 and noticed the regression.

The workaround from @bosd worked on my box - but not sure that’s really optimal, as @MaximilianGaedig states in his comment

matzew on Sep 18, 2023

Running into this problem on a new Fedora machine I’m trying to set up, except plain podman run is working fine, but docker-compose up is giving me this problem.

Podman 4.6.2, Compose v2.21.0

$ podman run --rm hello-world
!... Hello Podman World ...!
<snip>

$ cat docker-compose.yml
version: "3"

services:
  foo:
    image: hello-world

$ docker-compose up
[+] Running 2/2
 ✔ Network test_default   Created                                                                                  0.0s
 ✔ Container test-foo-1  Created                                                                                  1.0s
Attaching to test-foo-1
Error response from daemon: crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied

Downgrading to Podman 4.4.2 (only other installable version out of the box with dnf) shows no error (but just doesn’t have any output or anything weirdly), another server running 4.6.1 works perfectly fine.

Ovyerus on Sep 16, 2023

I know this is closed and all, but what worked for me was setting this environment variable as mentioned in the release notes:

DOCKER_BUILDKIT=0

podman version 4.6.1 Docker Compose version v2.21.0

NicolaiLolansen on Sep 15, 2023

Hi there, After install docker, I was able to run podman containers

sudo dnf -y install dnf-plugins-core sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo

sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo systemctl start docker

edgarjoao on Dec 6, 2023

Could this be due (also) to ACLs ? Specifically in my case, I had to set ZFS acltype=posix in order to NOT encounter issues while trying to spin up e.g. Mosquitto MQTT Container (otherwise I’d get mkdir - operation not permitted). See https://github.com/containers/podman/issues/11213.

However, now that ACLs are enabled in ZFS, it became much slower. I tried to set xattr=sa to make it faster by storing extended attributes in the inode/dnode directly, but that didn’t really help.

podman start mycontainer or even podman ps hangs indefinitively. Only solution is to reboot !

Then the hanging/freezing stopped and now I get this

[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

I also tried the previous fixes, both in /etc/systemd/system/user@.service and /etc/systemd/system/user@.service.d/override.conf but it doesn’t help.

I have a much older version of Podman being on Debian Linux but still … It also seems after enabling ACL I get more and more weird errors

host:
  arch: arm64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_arm64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 98.58
    systemPercent: 0.55
    userPercent: 0.87
  cpus: 8
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: Rock5B-01
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.2-1-arm64
  linkmode: dynamic
  logDriver: journald
  memFree: 15235072000
  memTotal: 16477868032
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+b1_arm64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1002/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_arm64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 10m 58.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.10-1_arm64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.10
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /home/podman/storage
  graphRootAllocated: 1931117854720
  graphRootUsed: 1314783232
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1002/containers
  volumePath: /home/podman/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/arm64
  Version: 4.3.1

Strangely it doesn’t happen with all containers. I could get (again) podman-compose to bring down (destroy) & up (recreate) an instance of Home Assistant, while Mosquitto MQTT Container Image stubbornly refuses to work.

luckylinux on Nov 30, 2023

I have updated to Podman 4.6.2 and noticed the regression. The workaround from @bosd worked on my box - but not sure that’s really optimal, as @MaximilianGaedig states in his comment

chmod: changing permissions of ‘/usr/lib/systemd/system/user@.service’: Read-only file system. Sudo doesn’t work, how do we write this file to remove that property ? Just mount directory, then change permissions

I fixed it below the link(https://github.com/containers/podman/issues/19930#issuecomment-1732162092). You can try it.

stan-shih on Sep 24, 2023

4 years later? The old bug should be locked and a new bug should be opened 🙏

jorhett on Sep 22, 2023

Update: This solved it for me:

edit /usr/lib/systemd/system/user@.service and remove the following line.

OOMScoreAdjust=100 [systemd/systemd@ce7de0b#diff-

Reboot

Source: https://bbs.archlinux.org/viewtopic.php?pid=2012166#p2012166

MacOS with latest podman here. I remove provided line via podman machine ssh connection however after restart of the VM, this property of OOMScoreAdjust=100 appears to be there again and error expectedly repeats, is there any workaround in that case? Thank you

setofaces on Sep 19, 2023

BTW this is tracked in #19930 and https://github.com/containers/podman/pull/19843 is supposed to fix it. (downgrading crun to 1.8.7 apparently also helps)

rhafer on Sep 18, 2023

Having this issue on Fedora Silverblue using toolbox, strangely didn’t happen until I rebased by system to KDE using this guide https://communityblog.fedoraproject.org/rebasing-fedora-silverblue-to-kinoite/ Doing it as sudo works fine however but that seems like it’s begging for trouble

edit: fixed itself after a reboot I will keep this comment in case someone can reproduce it using those steps

nonetrix on Jun 26, 2022

that is correct 🙂

haircommander on Oct 31, 2019