cli: dockerd: failed to start daemon: Devices cgroup isn't mounted

I have the same issue. However I found a workaround:

service docker stop
service containerd stop
cgroupfs-umount
cgroupfs-mount
service containerd start
service docker start

Sysinfo:

Linux vr 5.2.0-2-amd64 #1 SMP Debian 5.2.9-2 (2019-08-21) x86_64 GNU/Linux
cgroupfs-mount ver 1.4

containerd --version

containerd containerd.io 1.2.6 894b81a4b802e4eb2a91d1ce216b8817763c29fb

docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfca03
 Built:             Thu Aug 29 05:29:49 2019
 OS/Arch:           linux/amd64
 Experimental:      false

dnf install -y grubby

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

reboot

Hi All,

 Seems issue still persist on latest Fedora 31, even with adding extra command  

–exec-opt native.cgroupdriver=systemd

And per upstream workaround, which works temporarily with cgroup v1.

However, the question lays on how do we address/fix this issue permanently as it's kernel API related.

Thanks

Found the “bug”…

I forgot to mention in my previous comment that I use(d) systemd version 243. With systemd 242 works everything flawlessly… 😃

From the systemd changelog:

        * systemd now defaults to the "unified" cgroup hierarchy setup during
          build-time, i.e. -Ddefault-hierarchy=unified is now the build-time
          default. Previously, -Ddefault-hierarchy=hybrid was the default. This
          change reflects the fact that cgroupsv2 support has matured
          substantially in both systemd and in the kernel, and is clearly the
          way forward. Downstream production distributions might want to
          continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for
          their builds as unfortunately the popular container managers have not
          caught up with the kernel API changes.

Sooo… Houston, we have a problem:

1. systemd will (or already did) jump on the cgroupsv2 bandwagon…
2. cgroupfs-mount tools does not work with newer systemd setups.

“Same” issue in kubernetes

This has been making my Fedora 30->31 upgrades ugly. Of course Podman works in anti-competitive efforts to push out Docker. A system-wide kernel boot option shouldn’t be required for backwards compatibility with a container engine. Hmmm

Running this on Fedora 31, after an in-place upgrade from Fedora 30, works:

sudo grubby --args='systemd.unified_cgroup_hierarchy=0' --update-kernel /boot/vmlinuz-5.3.16-300.fc31.x86_64

See a specific comment on the Fedora 31 bug report and the Fedora 31 documentation on using grubby.

Very old fedora install here. I:

1. Added GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0" to /etc/default/grub
2. Ran grub2-mkconfig -o /boot/grub2/grub.cfg
3. Ran reboot

and everything was happy

@kj54321 Fixed by the workaround suggested, just add the argument systemd.unified_cgroup_hierarchy=0 in grub

Following also worked …

dnf install -y grubby

grubby --update-kernel=ALL --args=“systemd.unified_cgroup_hierarchy=0”

reboot

Obviously docker just has to support cgroupsv2.

From the cited distribution bug (which is a downstream workaround, not an upstream workaround):

cgroups v2 isn't exactly new or a surprise to anyone. Docker just dropped a ball as an upstream.

None of these comments are helpful. This is a community effort. Please keep politics out of it.

Update: installed the community edition and it works now with the added boot argument.

I added the argument but it did not solve the issue on a Fedora 31 install (upgraded from Fedora 30).

Here is my boot command line /proc/cmdline BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.3.11-300.fc31.x86_64 root=/dev/mapper/fedora_localhost--live-root ro resume=/dev/mapper/fedora_localhost--live-swap rd.lvm.lv=fedora_localhost-live/root rd.lvm.lv=fedora_localhost-live/swap rhgb quiet systemd.unified_cgroup_hierarchy=0

Based on https://github.com/docker/cli/issues/2104#issuecomment-563722141, the following worked for me on Ubuntu 22.04 as well (trying to run Docker-in-Docker with an old Docker 18.09.9 version).

• Edited GRUB_CMDLINE_LINUX to include systemd.unified_cgroup_hierarchy=0 in /etc/default/grub
• Ran sudo grub-mkconfig -o /boot/grub/grub.cfg
• Ran sudo reboot

I gave an update 12 days ago.

Will it work OOTB on Fedora 32? Not unless Fedora switches back to cgroups v1.

Support for cgroups v2 in Docker is still being worked on across the stack.

@jjspace This will not be supported until containerd 1.4 is released. I think we have all the bits we need in dockerd for this right now, but again this depends on containerd 1.4. There are also some outstanding pieces in runc.

We are currently planning the next major release of Docker (“20.03” is the placeholder version). I do not think we will be shipping that with containerd 1.4 since that is not ready yet… but in theory one containerd 1.4 is out once that’s upgraded (along with runc) it should work.