moby: Unable to run systemd in docker with ro /sys/fs/cgroup after systemd 248 host upgrade

remove ro flag from docker run arg -v /sys/fs/cgroup:/sys/fs/cgroup:ro

It didn’t help. I’m running Ubuntu 21.10 (Impish Indri).

For reference, it is possible with namespace isolation.

@skast96, it didn’t help either. I edited /etc/docker/daemon.json:

{"userns-remap": "default"}

Restarted docker. The dockremap user was created, as were the entries in /etc/sub{uid,gid}. The /var/lib/docker/100000.100000 dir was created. docker image ls produced no output. Then:

$ docker run -it --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup jrei/systemd-ubuntu
systemd 245.4-4ubuntu3.16 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to Ubuntu 20.04.4 LTS!

Set hostname to <1bdd4443336d>.
Failed to create /init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Exiting PID 1...

So the only workaround is supposedly to switch to the cgroup v1 mode (systemd.unified_cgroup_hierarchy=0):

• /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0"

• update-grub
• reboot

UPD And --cgroupns=host + -v /sys/fs/cgroup:/sys/fs/cgroup (w/o :ro), e.g.:

$ docker run -it --cgroupns=host --tmpfs /tmp --tmpfs /run --tmpfs /run/lock \
    -v /sys/fs/cgroup:/sys/fs/cgroup jrei/systemd-ubuntu

I have discovered two additional workarounds for this issue that effectively retain all features of unified cgroupv2 while maintaining security - no need for the --privileged flag and no access to the root of cgroupv2 hierarchy:

1. Use the --cgroupns host Docker option and a cgroupv2 sub-hierarchy volume binding for the container. Here is an example command:

# docker run --rm --name freeipa -it --read-only --security-opt seccomp=unconfined --hostname freeipa.corp --init=false --cgroupns host -v /sys/fs/cgroup/freeipa.scope:/sys/fs/cgroup:rw freeipa/freeipa-server:almalinux-9
systemd 252-13.el9_2 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.
Initializing machine ID from random generator.
Queued start job for default target Minimal target for containerized FreeIPA server.
[..]

Not perfect, next option is better IMO.

1. Mount /sys/fs/cgroup on the host without the nsdelegate mount option. Although there isn’t an explicit option to disable nsdelegate like nodiscard for discard (see link 1, link 2 for more information), there is a workaround. Simply run any container using Docker with the --cgroupns host option and without any cgroup volume bindings. For example:

# grep cgroup /proc/mounts 
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
# docker run --rm --cgroupns host ubuntu:latest echo done
done
# grep cgroup /proc/mounts 
cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime 0 0

After implementing these steps, you can run a container with Docker using --cgroupns private flag and volume binding of cgroupv2 sub-hierarchy. For example:

# docker run --rm --name freeipa -it --read-only --security-opt seccomp=unconfined --hostname freeipa.corp --init=false --cgroupns private -v /sys/fs/cgroup/freeipa.scope:/sys/fs/cgroup:rw freeipa/freeipa-server:almalinux-9
systemd 252-13.el9_2 running in system mode (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization container-other.
Detected architecture x86-64.
Initializing machine ID from random generator.
Queued start job for default target Minimal target for containerized FreeIPA server.
[..]

Please note that the information provided above applies specifically to CentOS Stream release 9 with kernel-ml-6.3.7-1.el9.elrepo, systemd-252.4-598.13.hs.el9 (Hyperscale SIG) and docker-ce-24.0.2-1 (systemd cgroup driver) although may help with a wide range of different scenarios.

Related: https://serverfault.com/questions/1053187/systemd-fails-to-run-in-a-docker-container-when-using-cgroupv2-cgroupns-priva/1054414#1054414

@marco-a-itl

This is the mount shown in the container:

# findmnt | grep cgroup
│ └─/sys/fs/cgroup      cgroup      cgroup2  rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot

This is ok (mode is rw). However I assume that you obtained this result with userns-remapping.

I think that it should be possible to have the same result without such daemon option, with the proper modifications on the docker engine, like podman does.

1. Use the --cgroupns host Docker option and a cgroupv2 sub-hierarchy volume binding for the container. Here is an example command:

docker run --rm --name freeipa -it --read-only --security-opt seccomp=unconfined --hostname freeipa.corp --init=false --cgroupns host -v /sys/fs/cgroup/freeipa.scope:/sys/fs/cgroup:rw freeipa/freeipa-server:almalinux-9

Thanks. This helped me too in starting a docker container with systemd inside (Fedora 37 host with cgroupv2).

I needed to add to daemon.json (and create the dockuser user on the docker host):

{
    "userns-remap": "dockuser"
}

I left out some of the options you used though:

docker run \
-it \
--rm \
--name ubuntu_systemd_local \
--tmpfs /tmp \
--tmpfs /run \
--tmpfs /run/lock \
--cgroupns private \
ubuntu_systemd:local

I used this Dockerfile: (I created encrypted_password with mkpasswd -m sha512crypt ‘password’)

FROM ubuntu:22.04
ENV DEBIAN_FRONTEND noninteractive
RUN yes | unminimize && \
echo 'root:_encrypted_password_' | chpasswd -e && \
sed -i -e 's/archive.ubuntu/fi.archive.ubuntu/g' /etc/apt/sources.list && \
apt-get -y update && \
apt-get -y install apt-utils && \
apt-get -y install dialog && \
apt-get -y install iputils-ping bind9-host iproute2 netcat-openbsd && \
apt-get -y install systemd dbus dbus-user-session dbus-x11 dconf-cli && \
apt-get -y install vim less nmon glances iptraf-ng \
cifs-utils elinks elinks-data \
irssi lftp mc mc-data unrar nmap ctorrent iotop powertop \
w3m radvd caca-utils httpie jq firejail curl nmap stress-ng \
cksfv mtr htop smem gddrescue oidentd ntpdate sysfsutils \
cpulimit expect stress-ng pavucontrol rtorrent screen telnet \
cabextract youtube-dl sshuttle emacs nethogs alien \
exfatprogs p7zip mosh keepassxc virt-what fdisk && \
curl -s https://packagecloud.io/install/repositories/ookla/speedtest-cli/script.deb.sh | bash && \
apt-get -y install speedtest
STOPSIGNAL SIGRTMIN+3
CMD [ "/sbin/init" ]

@x-yuri the docker approach is not working that great tbh. It is working with namespace isolation when creating a extra slice for docker and adding this slice to the docker run command like so:

docker run -it \
    --cgroup-parent=docker.slice \
    --cgroupns private \
    --tmpfs /tmp \
    --tmpfs /run \
    --tmpfs /run/lock \
    mySystemdImage:latest 

That kinda worked for me. However our other containers stopped working with namespace isolation because they were not configured for that. That meant to much work in order to run one container with systemd.

So I suggest you to just install podman. I experienced no drawbacks on my Arch Linux when having both docker and podman installed. Even the commands are the same. You would start your systemd container like that below with podman.

podman run -it mySystemdImage:latest 

Is there already a fix for this?

For reference, it is possible with namespace isolation. https://docs.docker.com/engine/security/userns-remap/ Or simply install podman.

Same here It was working with 247