podman: After podman 2 upgrade, systemd fails to start in containers on cgroups v1 hosts

@c-goes - see https://github.com/containers/podman/issues/7441. I think you’re hitting that.

if reverting (#6569) solves your issue, you can force a new scope wrapping podman with systemd-run as systemd-run --user --scope podman ....

In your case it will be: systemd-run --user --scope podman run -it jrei/systemd-ubuntu:16.04

OK that is just really cool. This is how I would implement.

As root: (likely wrap this up in a systemd unit to run at boot time, maybe systemd-cgv1.service)

mkdir /sys/fs/cgroup/systemd
mount -t cgroup cgroup -o none,name=systemd,xattr /sys/fs/cgroup/systemd

And wrap this up in something like systemd-cgv1-user@.service:

mkdir -P /sys/fs/cgroup/systemd/user.slice/user-1000.slice
chown -R 1000.1000 /sys/fs/cgroup/systemd/user.slice/user-1000.slice

Since we’re making our own convention it could even use username instead of uid…

Note this essentially creates a kinda unified hybrid… Host systemd doesn’t know about the v1 cgroup, which I suppose is fine.

$ cat /proc/self/cgroup
1:name=systemd:/
0::/user.slice/user-1000.slice/session-16.scope

So, as user - add this to your .bash_profile (or similar):

echo $BASHPID > "/sys/fs/cgroup/systemd/user.slice/user-$UID.slice/cgroup.procs"

And then you’re good to go. podman run --rm -it --annotation run.oci.systemd.force_cgroup_v1=/sys/fs/cgroup jrei/systemd-ubuntu:16.04

That works for me. That’s pretty cool @giuseppe

I wasn’t aware of the issue as I could have documented it better, but it seems that it is necessary to mount the named hierarchy in the host first:

# mkdir /sys/fs/cgroup/systemd && mount -t cgroup cgroup -o none,name=systemd,xattr /sys/fs/cgroup/systemd

also please enforce the systemd mode with --systemd always unless your init binary is /sbin/init or systemd

If you create a subcgroup like:

# mkdir /sys/fs/cgroup/systemd/1000
# chown 1000:1000 mkdir /sys/fs/cgroup/systemd/1000
# echo $ROOTLESS_TERMINAL_PROCESS_PID > /sys/fs/cgroup/systemd/1000/cgroup.procs

you’ll be able to use the feature also as rootless (user 1000 assumed in my example above).

You need to be root. It is a privileged operation

Known Workarounds

1. Enable cgroupsv2. Update /etc/default/grub to add systemd.unified_cgroup_hierarchy=1 to GRUB_CMDLINE_LINUX_DEFAULT and follow standard procedures to update Grub and reboot.
2. Don’t use images with old versions of systemd, such as Ubuntu 16.04.

But to be clear, you tested this on Cgroups V2, and podman 1.9.3 can’t run Systemd from Ubuntu 16.04 on Cgroups V2 either.

Did you test 16.04 on cgroups 1?

I show on Cgroups v1, you CAN run 16.04 images and old systemd images. It requires the same workarounds as before, systemd-run or whatever, to work on podman v2.

So ultimately.

1. Enable cgroupsv2. In your case, you do that with grub config, in FCOS, use kargs, etc. In F32, it’s already default. Realize when you enable Cgroups V2, you lose the ability to run OLD systemd, in any form, in any container. Unless @giuseppe 's patch to crun, dev, allows this. That said, you can happily use 16.04 for whatever you want. The limitation is that systemd didn’t get anything involving cgroups v2 until at least v233.

2. Launch your podman in a proper systemd scope, using systemd-run --scope --user or systemd-run --user -P

3. Manually specify cgroup-manager systemd:

(focal)mrwizard@FocalCG1Dev:~/src/podman
$ podman run --rm -it --cgroup-manager systemd jrei/systemd-ubuntu:16.04
systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Ubuntu 16.04.6 LTS!

Set hostname to <91b5f3e21931>.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to install release agent, ignoring: Permission denied
[  OK  ] Created slice System Slice.
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Slices.

If you want to be able to do sshd without systemd… you can. podman --init --rm -it ubuntu:16.04 sh -c 'apt-get update && apt-get install -y openssh-server && /usr/sbin/sshd -D'

Obviously you’d do that in your build, not on the command line. You could also use podman exec instead of ssh. We don’t know your use case, so we can’t advise better.

NOW, if someone who knows more than me wants to figure out WHY --cgroup-manager systemd is required, why podman isn’t automatically detecting that as an option or using it… those are actionable.

I can also say that setting

[engine]

# Cgroup management implementation used for the runtime.
# Valid options “systemd” or “cgroupfs”
#
cgroup_manager = "systemd"

Makes no difference, and the only warning I see cgroup related is: time=“2020-07-02T13:51:15-04:00” level=warning msg=“Failed to add conmon to cgroupfs sandbox cgroup: error creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/libpod_parent: permission denied”

Which isn’t entirely unexpected on CGv1. But the big question is - WHY is podman using cgroupfs?

Cgroups output when running podman from console:

12:hugetlb:/
11:freezer:/
10:blkio:/user.slice
9:cpu,cpuacct:/user.slice
8:perf_event:/
7:memory:/user.slice/user-1000.slice/session-232.scope
6:cpuset:/
5:pids:/user.slice/user-1000.slice/session-232.scope
4:devices:/user.slice
3:rdma:/
2:net_cls,net_prio:/
1:name=systemd:/user.slice/user-1000.slice/session-232.scope
0::/user.slice/user-1000.slice/session-232.scope

File system permissions:

total 0
drwxr-xr-x  6 root    root    0 kesä   28 16:27 .
drwxr-xr-x  3 root    root    0 kesä   28 16:27 ..
-rw-r--r--  1 root    root    0 kesä   28 20:18 cgroup.clone_children
-rw-r--r--  1 root    root    0 kesä   28 20:18 cgroup.procs
-rw-r--r--  1 root    root    0 kesä   28 20:18 notify_on_release
drwxr-xr-x  2 root    root    0 kesä   28 20:10 session-232.scope
drwxr-xr-x  2 root    root    0 kesä   28 20:18 session-2.scope
-rw-r--r--  1 root    root    0 kesä   28 20:18 tasks
drwxr-xr-x 44 someuser someuser 0 kesä   28 20:04 user@1000.service
drwxr-xr-x  2 root    root    0 kesä   28 20:18 user-runtime-dir@1000.service

I think I figured out why we are seeing different behavior. My cgroups looks following:

12:hugetlb:/
11:freezer:/
10:blkio:/user.slice
9:cpu,cpuacct:/user.slice
8:perf_event:/
7:memory:/user.slice/user-1000.slice/user@1000.service
6:cpuset:/
5:pids:/user.slice/user-1000.slice/user@1000.service
4:devices:/user.slice
3:rdma:/
2:net_cls,net_prio:/
1:name=systemd:/user.slice/user-1000.slice/user@1000.service/apps.slice/apps-org.gnome.Terminal.slice/vte-spawn-6773d329-9ee1-450e-ae44-d5e4810e64a2.scope/ef312c6459eb19dc7f99f918f0eb90c7a231c59f50e40e9141018baa13c48b35
0::/user.slice/user-1000.slice/user@1000.service/apps.slice/apps-org.gnome.Terminal.slice/vte-spawn-6773d329-9ee1-450e-ae44-d5e4810e64a2.scope

As you can see, I’m running my commands using gnome / xterm. Now, if I drop to console mode, I see exactly same behavior:

systemd v243.8-1.fc31 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization podman.
Detected architecture x86-64.

Welcome to Fedora 31 (Container Image)!

Set hostname to <9d16995357a6>.
Initializing machine ID from random generator.
Failed to create /user.slice/user-1000.slice/session-232.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

@skorhone Notice some differences in our different cases:

• podman 1.9.3 working: Detected virtualization docker.
• podman 2.0.0 not working: Detected virtualization container-other.
• podman 2.0.0 working: Detected virtualization podman.

Could that be related?