moby: `systemctl docker start` get stuck in cloud-init

I’ve come across something similar which is related to #41297 This is on Ubuntu 20.04 with docker-ce 20.10

If a systemd unit relies on docker and is set to install on multi-user.target it will never start

Sample systemd file

[Unit]
Requires=docker.service
After=docker.service

[Service]
ExecStart=/usr/bin/docker run nginx

[Install]
WantedBy=multi-user.target

The problem is that docker is set to start after multi-user.target

The docker systemd unit has:

WantedBy=multi-user.target

Which tells systemd that when multi-user.target starts docker.service should start

It also has: After=network-online.target firewalld.service containerd.service multi-user.target

Which tells systemd that docker should start after multi-user.target has fully started

These are conflicting we’re saying “multi-user.service depends on docker but docker must run after multi-user.service”

In the case of the sample unit it won’t start because it starts when multi-user.target starts but can’t because it required docker to be running which won’t start until after multi-user.target starts.

I suggest reverting 0ca7456e5284d4aa9f3e37e69c7c93eff4420d3d I can’t think of a better solution at the moment and if you search the internet for start docker container with systemd you’ll find a ton of unit files which look like the sample so this has the potential to break things for a lot of people.

The docker 20.10.1 packages are now available on download.docker.com; make sure to switch back repositories to the main (download.docker.com) package repo if you temporarily switched to the staging repository.

I’m closing this issue, as this should be resolved, but feel free to continue the conversation.

20.10 changed the following line in docker.service.

After=network-online.target firewalld.service containerd.service multi-user.target

The last entry was added in 20.10.

After I removed multi-user.target from the After=..., both docker.service and the docker container defined in another service unit started without problems. See docker/for-linux#1162

@zigarn or anyone hitting this issue would you mind running:

curl -fsSL https://get.docker.com/ | DOWNLOAD_URL=https://download-stage.docker.com REPO_FILE=docker-ce-staging.repo sh

And confirm you no longer see the issue?

@tiborvass I confirm that the issue is gone with staging package.

is this a unit you defined yourself, or is that part of some package? (curious what package the conflict is in)

One I wrote the one I supplied is the simplest form. You’ll find lots of similar ones on the internet as a guide.

The wants/wanted by/after etc. relationship is complicated.

The gist is that after determines the order where as wants determines the dependencies. This is why it’s suggested to use both.

So for example if you want a service to only start once docker is ready you use

Requires=docker.service
After=docker.service

The Require means start docker when the services starts. The After means start the service once docker has fully started.

Wants is just a less strict version of requires (it doesn’t cause failure)

So WantedBy and After of the same service doesn’t make a lot of sense.