podman: podman 1.9.0 rootless on CentOS error: cgroup namespaces aren't enabled in the kernel: OCI runtime error
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Fresh CentOS 7.7 kvm-guest w/ Kubic CentOS7 repo added, installs podman 1.9.0 however no containers can be started:
Steps to reproduce the issue:
-
fresh centos7 installed
-
setup kubic centos7 libcontainers-stable repo
-
launch nginx container
Describe the results you received:
[test@test ~]$ podman run -d -p 8080:80 --name nginx docker.io/library/nginx:alpine
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
Trying to pull docker.io/library/nginx:alpine...
Getting image source signatures
Copying blob b14da7a62044 done
Copying blob aad63a933944 done
Copying config 29b49a39bc done
Writing manifest to image destination
Storing signatures
Error: cgroup namespaces aren't enabled in the kernel: OCI runtime error
Describe the results you expected:
container launching normally
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
[test@test ~]$ podman version
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
Version: 1.9.0
RemoteAPI Version: 1
Go Version: go1.13.6
OS/Arch: linux/amd64
Output of podman info --debug:
[test@test ~]$ podman info --debug
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
debug:
compiler: gc
gitCommit: ""
goVersion: go1.13.6
podmanVersion: 1.9.0
host:
arch: amd64
buildahVersion: 1.14.8
cgroupVersion: v1
conmon:
package: conmon-2.0.15-2.1.el7.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.15, commit: 78140cdc1001a632e6d9cd477ae2cfbc49927bdf'
cpus: 1
distribution:
distribution: '"centos"'
version: "7"
eventLogger: journald
hostname: test
idMappings:
gidmap:
- container_id: 0
host_id: 100
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 3.10.0-1062.18.1.el7.x86_64
memFree: 1636225024
memTotal: 1927213056
ociRuntime:
name: runc
package: runc-1.0.0-15.1.el7.x86_64
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc10
commit: 67b92f062188d9cb6472b428855432c9f35efcf5
spec: 1.0.1-dev
os: linux
rootless: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.0.0-4.1.el7.x86_64
version: |-
slirp4netns version 1.0.0
commit: a3be729152a33e692cd28b52f664defbf2e7810a
libslirp: 4.2.0
swapFree: 0
swapTotal: 0
uptime: 6m 15.53s
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- registry.centos.org
- docker.io
store:
configFile: /home/test/.config/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 0
stopped: 1
graphDriverName: vfs
graphOptions: {}
graphRoot: /home/test/.local/share/containers/storage
graphStatus: {}
imageStore:
number: 1
runRoot: /run/user/1000/containers
volumePath: /home/test/.local/share/containers/storage/volumes
Package info (e.g. output of rpm -q podman or apt list podman):
[test@test ~]$ rpm -qi podman
Name : podman
Epoch : 0
Version : 1.9.0
Release : 1.1.el7
Architecture: x86_64
Install Date: Tue Apr 21 23:24:12 2020
Group : Unspecified
Size : 78805972
License : ASL 2.0
Signature : RSA/SHA256, Thu Apr 16 15:54:39 2020, Key ID 4d64390375060aa4
Source RPM : podman-1.9.0-1.1.el7.src.rpm
Build Date : Thu Apr 16 15:54:18 2020
Build Host : lamb28
Additional environment details (AWS, VirtualBox, physical, etc.):
From a fresh CentOS 7 host - here's the full session of commands for your review, from adding the repo to installing the packages fresh, to logging in as a non-root user and attempting to run a container
$ ssh root@192.168.0.98
root@192.168.0.98's password:
Last login: Mon Apr 20 15:57:44 2020 from 192.168.0.52
[root@test ~]# yum upgrade
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.ukfast.co.uk
* extras: mozart.ee.ic.ac.uk
* updates: mirrors.ukfast.co.uk
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
No packages marked for update
[root@test ~]# sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/devel:kubic:libcontainers:stable.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 357 100 357 0 0 687 0 --:--:-- --:--:-- --:--:-- 687
[root@test ~]# sudo yum -y install podman
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.ukfast.co.uk
* extras: mozart.ee.ic.ac.uk
* updates: mirrors.ukfast.co.uk
devel_kubic_libcontainers_stable | 1.3 kB 00:00:00
devel_kubic_libcontainers_stable/primary | 10 kB 00:00:00
devel_kubic_libcontainers_stable 31/31
Resolving Dependencies
--> Running transaction check
---> Package podman.x86_64 0:1.9.0-1.1.el7 will be installed
--> Processing Dependency: podman-plugins = 1.9.0-1.1.el7 for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: slirp4netns >= 0.3.0-2 for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: containernetworking-plugins >= 0.7.5-1 for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: runc for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: nftables for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: containers-common for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: container-selinux for package: podman-1.9.0-1.1.el7.x86_64
--> Processing Dependency: conmon for package: podman-1.9.0-1.1.el7.x86_64
--> Running transaction check
---> Package conmon.x86_64 2:2.0.15-2.1.el7 will be installed
---> Package container-selinux.noarch 2:2.107-3.el7 will be installed
--> Processing Dependency: selinux-policy-targeted >= 3.13.1-216.el7 for package: 2:container-selinux-2.107-3.el7.noarch
--> Processing Dependency: selinux-policy-base >= 3.13.1-216.el7 for package: 2:container-selinux-2.107-3.el7.noarch
--> Processing Dependency: selinux-policy >= 3.13.1-216.el7 for package: 2:container-selinux-2.107-3.el7.noarch
--> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch
---> Package containernetworking-plugins.x86_64 0:0.8.5-145.1.el7 will be installed
---> Package containers-common.x86_64 2:0.2.0-2.1.el7 will be installed
---> Package nftables.x86_64 1:0.8-14.el7 will be installed
--> Processing Dependency: libnftnl.so.7(LIBNFTNL_5)(64bit) for package: 1:nftables-0.8-14.el7.x86_64
--> Processing Dependency: libnftnl.so.7()(64bit) for package: 1:nftables-0.8-14.el7.x86_64
---> Package podman-plugins.x86_64 0:1.9.0-1.1.el7 will be installed
---> Package runc.x86_64 2:1.0.0-15.1.el7 will be installed
--> Processing Dependency: criu for package: 2:runc-1.0.0-15.1.el7.x86_64
---> Package slirp4netns.x86_64 0:1.0.0-4.1.el7 will be installed
--> Processing Dependency: libslirp.so.0(SLIRP_4.1)(64bit) for package: slirp4netns-1.0.0-4.1.el7.x86_64
--> Processing Dependency: libslirp.so.0(SLIRP_4.0)(64bit) for package: slirp4netns-1.0.0-4.1.el7.x86_64
--> Processing Dependency: libslirp.so.0()(64bit) for package: slirp4netns-1.0.0-4.1.el7.x86_64
--> Running transaction check
---> Package criu.x86_64 0:3.12-2.el7 will be installed
--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.12-2.el7.x86_64
--> Processing Dependency: libnl-3.so.200(libnl_3)(64bit) for package: criu-3.12-2.el7.x86_64
--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.12-2.el7.x86_64
--> Processing Dependency: libnl-3.so.200()(64bit) for package: criu-3.12-2.el7.x86_64
--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.12-2.el7.x86_64
---> Package libnftnl.x86_64 0:1.0.8-1.el7 will be installed
--> Processing Dependency: libjansson.so.4()(64bit) for package: libnftnl-1.0.8-1.el7.x86_64
---> Package libslirp.x86_64 0:4.2.0-2.1.el7 will be installed
---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed
--> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libselinux-python for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libcgroup for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64
--> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64
---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be installed
---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be installed
--> Running transaction check
---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed
---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed
---> Package jansson.x86_64 0:2.10-1.el7 will be installed
---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed
---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed
---> Package libnl3.x86_64 0:3.2.28-4.el7 will be installed
---> Package libselinux-python.x86_64 0:2.5-14.1.el7 will be installed
---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed
---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed
---> Package python-IPy.noarch 0:0.75-6.el7 will be installed
---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================================================
Installing:
podman x86_64 1.9.0-1.1.el7 devel_kubic_libcontainers_stable 22 M
Installing for dependencies:
audit-libs-python x86_64 2.8.5-4.el7 base 76 k
checkpolicy x86_64 2.5-8.el7 base 295 k
conmon x86_64 2:2.0.15-2.1.el7 devel_kubic_libcontainers_stable 34 k
container-selinux noarch 2:2.107-3.el7 extras 39 k
containernetworking-plugins x86_64 0.8.5-145.1.el7 devel_kubic_libcontainers_stable 35 M
containers-common x86_64 2:0.2.0-2.1.el7 devel_kubic_libcontainers_stable 53 k
criu x86_64 3.12-2.el7 base 453 k
jansson x86_64 2.10-1.el7 base 37 k
libcgroup x86_64 0.41-21.el7 base 66 k
libnet x86_64 1.1.6-7.el7 base 59 k
libnftnl x86_64 1.0.8-1.el7 base 77 k
libnl3 x86_64 3.2.28-4.el7 base 278 k
libselinux-python x86_64 2.5-14.1.el7 base 235 k
libsemanage-python x86_64 2.5-14.el7 base 113 k
libslirp x86_64 4.2.0-2.1.el7 devel_kubic_libcontainers_stable 62 k
nftables x86_64 1:0.8-14.el7 base 186 k
podman-plugins x86_64 1.9.0-1.1.el7 devel_kubic_libcontainers_stable 2.3 M
policycoreutils-python x86_64 2.5-33.el7 base 457 k
protobuf-c x86_64 1.0.2-3.el7 base 28 k
python-IPy noarch 0.75-6.el7 base 32 k
runc x86_64 2:1.0.0-15.1.el7 devel_kubic_libcontainers_stable 4.5 M
selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k
selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M
setools-libs x86_64 3.3.8-4.el7 base 620 k
slirp4netns x86_64 1.0.0-4.1.el7 devel_kubic_libcontainers_stable 43 k
Transaction Summary
==============================================================================================================================================================================================================
Install 1 Package (+25 Dependent packages)
Total download size: 74 M
Installed size: 212 M
Downloading packages:
(1/26): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00
(2/26): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00
warning: /var/cache/yum/x86_64/7/devel_kubic_libcontainers_stable/packages/conmon-2.0.15-2.1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 75060aa4: NOKEY ] 0.0 B/s | 411 kB --:--:-- ETA
Public key for conmon-2.0.15-2.1.el7.x86_64.rpm is not installed
(4/26): conmon-2.0.15-2.1.el7.x86_64.rpm | 34 kB 00:00:01
(5/26): jansson-2.10-1.el7.x86_64.rpm | 37 kB 00:00:00
(6/26): libnet-1.1.6-7.el7.x86_64.rpm | 59 kB 00:00:00
(7/26): containers-common-0.2.0-2.1.el7.x86_64.rpm | 53 kB 00:00:00
(8/26): criu-3.12-2.el7.x86_64.rpm | 453 kB 00:00:00
(9/26): libnftnl-1.0.8-1.el7.x86_64.rpm | 77 kB 00:00:00
(10/26): libselinux-python-2.5-14.1.el7.x86_64.rpm | 235 kB 00:00:00
(11/26): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00
(12/26): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00
(13/26): nftables-0.8-14.el7.x86_64.rpm | 186 kB 00:00:00
(14/26): libslirp-4.2.0-2.1.el7.x86_64.rpm | 62 kB 00:00:00
(15/26): libnl3-3.2.28-4.el7.x86_64.rpm | 278 kB 00:00:01
(16/26): podman-1.9.0-1.1.el7.x86_64.rpm | 22 MB 00:00:03
(17/26): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00
(18/26): protobuf-c-1.0.2-3.el7.x86_64.rpm | 28 kB 00:00:00
(19/26): containernetworking-plugins-0.8.5-145.1.el7.x86_64.rpm | 35 MB 00:00:06
(20/26): podman-plugins-1.9.0-1.1.el7.x86_64.rpm | 2.3 MB 00:00:00
(21/26): selinux-policy-3.13.1-252.el7_7.6.noarch.rpm | 492 kB 00:00:00
(22/26): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:00
(23/26): runc-1.0.0-15.1.el7.x86_64.rpm | 4.5 MB 00:00:01
(24/26): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:01
(25/26): slirp4netns-1.0.0-4.1.el7.x86_64.rpm | 43 kB 00:00:00
(26/26): selinux-policy-targeted-3.13.1-252.el7_7.6.noarch.rpm | 7.0 MB 00:00:01
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 9.1 MB/s | 74 MB 00:00:08
Retrieving key from http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/repodata/repomd.xml.key
Importing GPG key 0x75060AA4:
Userid : "devel:kubic OBS Project <devel:kubic@build.opensuse.org>"
Fingerprint: 2472 d6d0 d2f6 6af8 7aba 8da3 4d64 3903 7506 0aa4
From : http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/repodata/repomd.xml.key
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : selinux-policy-3.13.1-252.el7_7.6.noarch 1/26
Installing : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/26
Installing : libcgroup-0.41-21.el7.x86_64 3/26
Installing : podman-plugins-1.9.0-1.1.el7.x86_64 4/26
Installing : libnl3-3.2.28-4.el7.x86_64 5/26
Installing : containernetworking-plugins-0.8.5-145.1.el7.x86_64 6/26
Installing : jansson-2.10-1.el7.x86_64 7/26
Installing : libnftnl-1.0.8-1.el7.x86_64 8/26
Installing : 1:nftables-0.8-14.el7.x86_64 9/26
Installing : libselinux-python-2.5-14.1.el7.x86_64 10/26
Installing : audit-libs-python-2.8.5-4.el7.x86_64 11/26
Installing : libnet-1.1.6-7.el7.x86_64 12/26
Installing : libsemanage-python-2.5-14.el7.x86_64 13/26
Installing : 2:conmon-2.0.15-2.1.el7.x86_64 14/26
Installing : protobuf-c-1.0.2-3.el7.x86_64 15/26
Installing : criu-3.12-2.el7.x86_64 16/26
Installing : setools-libs-3.3.8-4.el7.x86_64 17/26
Installing : libslirp-4.2.0-2.1.el7.x86_64 18/26
[... trunc since my scrollback buffer got overwritten... ]
[root@test ~]# echo user.max_user_namespaces=15000 > /etc/sysctl.d/95-user-namespaces.conf
[root@test ~]# reboot
[... host reboots ...]
$ ssh test@192.168.0.98
test@192.168.0.98's password:
Last login: Tue Apr 21 23:21:45 2020 from 192.168.0.52
[test@test ~]$ podman ps -a
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[test@test ~]$ podman run -d -p 8080:80 --name nginx docker.io/library/nginx:alpine
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
Trying to pull docker.io/library/nginx:alpine...
Getting image source signatures
Copying blob b14da7a62044 done
Copying blob aad63a933944 done
Copying config 29b49a39bc done
Writing manifest to image destination
Storing signatures
Error: cgroup namespaces aren't enabled in the kernel: OCI runtime error
whereas if we install podman 1.8.2 rpms back, this works fine:
[root@test ~]# rpm -Uvh --oldpackage podman-1.8.2-2.el7.x86_64.rpm podman-plugins-1.8.2-2.el7.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:podman-plugins-0:1.8.2-2.el7 ################################# [ 25%]
2:podman-0:1.8.2-2.el7 ################################# [ 50%]
Cleaning up / removing...
3:podman-0:1.9.0-1.1.el7 ################################# [ 75%]
4:podman-plugins-0:1.9.0-1.1.el7 ################################# [100%]
[root@test ~]# logout
Connection to 192.168.0.98 closed.
$ ssh test@192.168.0.98
test@192.168.0.98's password:
Last login: Tue Apr 21 23:25:44 2020 from 192.168.0.52
[test@test ~]$ podman rm nginx
6d6612ab42ce421a15018d3615682a179eb2b6179afcbe7b32c20d5c36abe59a
[test@test ~]$ podman image prune -af
docker.io/library/nginx:alpine
[test@test ~]$ podman run -d -p 8080:80 --name nginx docker.io/library/nginx:alpine
Trying to pull docker.io/library/nginx:alpine...
Getting image source signatures
Copying blob aad63a933944 done
Copying blob b14da7a62044 done
Copying config 29b49a39bc done
Writing manifest to image destination
Storing signatures
40edf62df3db3d585ea5252cfa45e9d19c3b02580455462efb4eddd085018770
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (15 by maintainers)
the fix solves the issue but I don’t think it is the correct fix.
We shouldn’t change the default for cgroupns. On cgroup v1 it should still default to host otherwise it is a breaking change.
I think we need to handle it from Podman. If the user doesn’t override it in the conf file, we need to default to
hoston cgroup v1 andprivateon cgroup v2.