podman: bash: error while loading shared libraries: libc.so.6: cannot change memory protections
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
After updating Fedora 33 to podman 3.1.2 I started getting this error whenever trying to run a container
Steps to reproduce the issue:
-
On a Fedora 33 system, update to podman 3.1.2
-
As a regular user, run
podman system resetto ensure a clean state -
As the same user, run
podman run -it --rm fedora:33(or alpine or basically any container that uses bash)
Describe the results you received:
bash: error while loading shared libraries: libc.so.6: cannot change memory protections
In /var/log/audit/audit.log:
type=AVC msg=audit(1624991277.062:935): avc: denied { read } for pid=15030 comm="bash" path="/usr/lib64/libc-2.32.so" dev="dm-10" ino=16779410 scontext=system_u:system_r:container_t:s0:c918,c946 tcontext=unconfined_u:object_r:data_home_t:s0 tclass=file permissive=0
Describe the results you expected:
A bash prompt
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
Version: 3.1.2
API Version: 3.1.2
Go Version: go1.15.11
Built: Tue May 11 09:53:47 2021
OS/Arch: linux/amd64
Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.20.1
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.0.27-2.fc33.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.27, commit: '
cpus: 12
distribution:
distribution: fedora
version: "33"
eventLogger: journald
hostname: localhost.localdomain
idMappings:
gidmap:
- container_id: 0
host_id: 4179
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 4179
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.12.12-200.fc33.x86_64
linkmode: dynamic
memFree: 26389209088
memTotal: 33388281856
ociRuntime:
name: crun
package: crun-0.19.1-3.fc33.x86_64
path: /usr/bin/crun
version: |-
crun version 0.19.1
commit: 1535fedf0b83fb898d449f9680000f729ba719f5
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/4179/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
selinuxEnabled: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.9-1.fc33.x86_64
version: |-
slirp4netns version 1.1.9
commit: 4e37ea557562e0d7a64dc636eff156f64927335e
libslirp: 4.3.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.0
swapFree: 8585732096
swapTotal: 8585732096
uptime: 34m 19.24s
registries:
docker.io:
Blocked: false
Insecure: false
Location: docker.io
MirrorByDigestOnly: false
Mirrors:
- Insecure: false
Location: mirror.gcr.io
Prefix: docker.io
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- registry.centos.org
- docker.io
store:
configFile: /home/cevich/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/cevich/.local/share/containers/storage
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 2
runRoot: /run/user/4179/containers
volumePath: /home/cevich/.local/share/containers/storage/volumes
version:
APIVersion: 3.1.2
Built: 1620741227
BuiltTime: Tue May 11 09:53:47 2021
GitCommit: ""
GoVersion: go1.15.11
OsArch: linux/amd64
Version: 3.1.2
Package info (e.g. output of rpm -q podman or apt list podman):
podman-3.1.2-2.fc33.x86_64
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)
No
Additional environment details (AWS, VirtualBox, physical, etc.):
Attempting to run the alpine container gives a different error message, but similar SELinux errors.
Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: Permission denied
Error relocating /bin/sh: RELRO protection failed: Permission denied
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 33 (11 by maintainers)
Commits related to this issue
- Fix CI in zuul For some (unknown) reason /home/zuul-worker/.local/share/containers selinux permissions are wrong and podman will not run properly. Creating the same dir before running podman with the... — committed to majamassarini/hardly by majamassarini 2 years ago
- Fix CI in zuul For some (unknown) reason /home/zuul-worker/.local/share/containers selinux permissions are wrong and podman will not run properly. Creating the same dir before running podman with the... — committed to majamassarini/hardly by majamassarini 2 years ago
- Merge pull request #60 from majamassarini/fix-zuul-ci Fix CI in zuul For some (unknown) reason /home/zuul-worker/.local/share/containers selinux permissions are wrong and podman does not run properl... — committed to packit/hardly by softwarefactory-project-zuul[bot] 2 years ago
restorecon -R -F $HOME/.local/share/containersfixed it, thank you!I’m on Fedora 38, Podman 4.5.0 and I no longer see this issue 🥳
Try restorecon -R -F $HOME/.local/share/containers
It looks like you relabeled the homedir ( $HOME/.local/share/containers) content while in a container.
Update: This problem has been observed recently in F35 as well. Though the ultimate cause is unknown at this time, the trouble comes from the same mechanism: Wrong SELinux type labels set on
$HOME/.local/share/containers/storage/overlay*(or $HOME/.local/share/containers in general). Generally speaking, theoverlay*sub-directories should be labeled with thecontainer_ro_file_ttype, with most/everything else beingdata_home_t.You can use
ls -Zto inspect the labels. If the labels are wrong they can be force reset using the (somewhat sledge-hammer) command:fixfiles restore $HOME/.local/share/containers. Or you can fuss with the labels in a more surgical fashion usingrestoreconorchcon(see man pages for details).As for the root-cause, certainly things like tarball or backup restores from earlier Fedora versions, could easily be blamed. Normally individual file/directory level label changes aren’t logged anywhere, as it would place an extraordinary burden on the logging/audit subsystem. So unless you have an incredible level of oversight, it can be difficult to determine the exact cause for the incorrect labels.
It was suggested that this is a known issue due to a necessary SELinux policy update. with a documented workaround:
Temporarily run in permissive mode (
sudo setenforce 0)