moby: Xenial (16.04) - Failed to start Docker Application Container Engine

The output indicates that there’s an issue with devicemapper. Some things I notice / wonder;

• Is this an upgrade from an earlier version of Ubuntu (and Docker)?
• Did you manually configure docker to use the devicemapper driver? The default on Ubuntu/xenial is aufs (but on recent kernels, overlay may be a better choice even)
• It looks like you’re running a custom kernel, because Xenial ships with kernel 4.4; https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Linux_kernel_4.4. Note that we only support kernels that are provided by your distro, not custom kernels

If you didn’t pick devicemapper for a reason, and don’t have any images/containers from a previous install that you want to keep, I suggest to;

• wipe /var/lib/docker (rm -rf /var/lib/docker) **this will remove all existing containers and images
• configure your daemon to use the “overlay” storage driver. you can do so by adding -s overlay to the daemon options (see https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options)

Solution is at the page indicated by @victor-ono

edit /etc/default/docker file and add the option: DOCKER_OPTS=“-s overlay”

then systemctl restart docker.service

Maybe this will help someone too. I’ve got a 4.5.7 kernel and Ubuntu Xenial (16.04)

The Docker installation process hang

root@scw-73792f:~# apt -y install docker-engine
...
Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
Setting up cgroupfs-mount (1.2) ...
Setting up libltdl7:amd64 (2.4.6-0.1) ...
Setting up docker-engine (1.12.3-0~xenial) ...
Progress: [ 89%]

Process backtrace

root     19943 19942  0 20:30 pts/1    00:00:00 /bin/systemd-tty-ask-password-agent --watch
root     19942 19908  0 20:30 pts/1    00:00:00 systemctl start docker.service
root     19908 19855  0 20:30 pts/1    00:00:00 /bin/sh /usr/sbin/invoke-rc.d docker start
root     19855 19740  0 20:30 pts/1    00:00:00 /bin/sh /var/lib/dpkg/info/docker-engine.postinst configure
root     19740 18060  0 20:30 pts/1    00:00:00 /usr/bin/dpkg --status-fd 24 --configure busybox-initramfs:amd64 cpio:amd64 initramfs-tools-bin:amd64 libklibc:amd64 klibc-utils:amd64 initramfs-tools-core:all linux-base:all initramfs-tools:all libapparmor-perl:amd64 apparmor:amd64 aufs-tools:amd64 cgroupfs-mount:all libltdl7:amd64 docker-engine:amd64 liberror-perl:all git-man:all git:amd64 patch:amd64
root     18060  4323  0 20:29 pts/0    00:00:01 apt -y install docker-engine

I have noticed the dockerd process was also running at that time:

             ├─docker.service
             │ ├─19945 /usr/bin/dockerd -H fd://
             │ └─19966 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc

But docker info commands were hanging too.

I have tried restarting docker service, which caused the installation to fail with the following output:

apt -y install docker-engine (from above)
...
Processing triggers for systemd (229-4ubuntu12) ...
Processing triggers for libc-bin (2.23-0ubuntu4) ...
Errors were encountered while processing:
 docker-engine
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@scw-73792f:~# echo $?
100

Then, each time I have been trying to continue the installation apt -f install, I was getting the same error. And each time I was trying to restart docker service manually, the following message was getting into /var/log/syslog file:

...
Nov  6 21:16:30 scw-73792f dockerd[20892]: time="2016-11-06T21:16:30.251654378Z" level=error msg="[graphdriver] prior storage driver \"devicemapper\" failed: devicemapper: Can't set task name /dev/mapper/docker-43:0-395135-pool"
...

So it made me thinking that something gone wrong with he devicemapper. Actually I could not remove the docker devicemapper loop devices using the losetup -D:

root@scw-73792f:~# losetup -a
/dev/loop0: [11008]:395139 (/var/lib/docker/devicemapper/devicemapper/data)
/dev/loop1: [11008]:395140 (/var/lib/docker/devicemapper/devicemapper/metadata)
root@scw-73792f:~# losetup -D
root@scw-73792f:~# losetup -a
/dev/loop0: [11008]:395139 (/var/lib/docker/devicemapper/devicemapper/data)
/dev/loop1: [11008]:395140 (/var/lib/docker/devicemapper/devicemapper/metadata)

The only thing that helped is dmsetup:

root@scw-73792f:~# dmsetup table
docker-43:0-395135-pool: 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing 
root@scw-73792f:~# dmsetup remove docker-43:0-395135-pool
root@scw-73792f:~# dmsetup table
No devices found
root@scw-73792f:~# losetup -a
root@scw-73792f:~# 

Then the installation could successfully be finished:

root@scw-73792f:~# apt -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up docker-engine (1.12.3-0~xenial) ...
root@scw-73792f:~# echo $?
0

Voila!

I need to mention, that on the moment of Docker installation, I did not have dmsetup installed at all. So I think that is the real cause of the problem.

Actually, without having dmsetup installed, neither cryptsetup luksOpen was working for me. Since it was hanging on semop(0, [{0, 0, 0}], 1) syscall due to missing udev rules /lib/udev/rules.d/55-dm.rules.

I think this was due to missing hooks which dmsetup is bringing with itself:

root@scw-73792f:~# dpkg -L dmsetup
...
/lib/udev/rules.d/55-dm.rules
/lib/udev/rules.d/60-persistent-storage-dm.rules
...

So maybe it would be making sense to add the dmsetup as a dependency in docker-engine .deb, what do you think, @thaJeztah ?

I managed to make it work by following the link @thaJeztah posted for the config file. As root, create a file /etc/docker/daemon.json with the following content:

{
    "storage-driver": "overlay2"
}

This works with systemd for me with Ubuntu 16.04 and stock 4.4 kernel.

Edit: updated per thaJeztah comment below

@misham for 4.4 kernels, I’d recommend using the overlay2 storage driver, which has the same advantages as overlay but not the limitations it has (i.e. running out of inodes)

@boussou that won’t work for systems using systemd (installed through the official packages); the easiest way, and something that always works, is to use a daemon configuration file. See https://docs.docker.com/engine/reference/commandline/dockerd/#/daemon-configuration-file

@thaJeztah If I just want to install docker-ce with my customized kernel, say, the kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/ . I’m now using the kernel:

jiapei@jiapei-GT72-6QE:~$ uname -r
4.14.5-041405-generic
jiapei@jiapei-GT72-6QE:~$ uname -a
Linux jiapei-GT72-6QE 4.14.5-041405-generic #201712101332 SMP Sun Dec 10 13:33:10 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

But I failed to install docker-ce with the following ERROR message:

jiapei@jiapei-GT72-6QE:~$ sudo apt install docker-ce
[sudo] password for jiapei: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
docker-ce is already the newest version (17.09.1~ce-0~ubuntu).
The following packages were automatically installed and are no longer required:
  bbswitch-dkms libjansson4 libxnvctrl0 screen-resolution-extra
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up docker-ce (17.09.1~ce-0~ubuntu) ...
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2017-12-17 19:22:06 CST; 5ms ago
     Docs: https://docs.docker.com
  Process: 15122 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 15122 (code=exited, status=1/FAILURE)

Dec 17 19:22:06 jiapei-GT72-6QE systemd[1]: Failed to start Docker Application Container Engine.
Dec 17 19:22:06 jiapei-GT72-6QE systemd[1]: docker.service: Unit entered failed state.
Dec 17 19:22:06 jiapei-GT72-6QE systemd[1]: docker.service: Failed with result 'exit-code'.
dpkg: error processing package docker-ce (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up python3-scipy (0.17.0-1) ...
[Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/scipy/_lib/decorator.py'
dpkg: error processing package python3-scipy (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 docker-ce
 python3-scipy
E: Sub-process /usr/bin/dpkg returned an error code (1)```

http://stackoverflow.com/questions/37227349/unable-to-start-docker-service-in-ubuntu-16-04

You’re welcome, and good to hear it worked out in the end!