moby: "docker daemon -H fd://" fails with message "No sockets found" under Ubuntu 15.10

Docker used to work fine, but after I ran the official PostgreSQL image (postgres:9.5.3), commands like docker ps, docker images and systemctl start docker would hang.

I had to kill -9 all docker-related processes and apt-get remove docker-engine, and then rm -rf /var/lib/docker. After that, I installed docker-engine via apt-get install docker-engine and also via get.docker.com, but I couldn’t get Docker to run again.


BUG REPORT INFORMATION

Output of docker version:

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:38:55 2016
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

Output of docker info:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

However, If I run Docker by typing docker daemon (instead of systemctl start docker), docker info gives this:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.2.0-34-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.56 GiB
Name: thundera
ID: KF4T:5CCO:DUDH:ECCB:3CYT:KMNJ:K7LI:LIXM:HZHP:WU74:ZK6C:YO5Y
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username: myuser
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.): Physical Ubuntu 15.10 Linux thundera 4.2.0-34-generic #39-Ubuntu SMP Thu Mar 10 22:13:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$> free -m
             total       used       free     shared    buffers     cached
Mem:         15934       3663      12270         76       2275        906
-/+ buffers/cache:        481      15452
Swap:        16269          0      16269

$> df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7,8G     0  7,8G   0% /dev
tmpfs           1,6G  9,3M  1,6G   1% /run
/dev/sda2       213G  6,9G  196G   4% /
tmpfs           7,8G     0  7,8G   0% /sys/fs/cgroup
/dev/sda1       511M  3,4M  508M   1% /boot/efi

Steps to reproduce the issue:

$> apt-get install docker-engine 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  docker-engine
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/14,5 MB of archives.
After this operation, 73,4 MB of additional disk space will be used.
Selecting previously unselected package docker-engine.
(Reading database ... 216726 files and directories currently installed.)
Preparing to unpack .../docker-engine_1.11.1-0~wily_amd64.deb ...
Unpacking docker-engine (1.11.1-0~wily) ...
Processing triggers for man-db (2.7.4-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (225-1ubuntu9) ...
Setting up docker-engine (1.11.1-0~wily) ...

$> docker daemon
INFO[0000] New containerd process, pid: 6963

WARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=65536
INFO[0001] Graph migration to content-addressability took 0.00 seconds 
INFO[0001] Firewalld running: false                     
INFO[0001] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
WARN[0001] Your kernel does not support swap memory limit. 
WARN[0001] mountpoint for pids not found                
INFO[0001] Loading containers: start.                   

INFO[0001] Loading containers: done.                    
INFO[0001] Daemon has completed initialization          
INFO[0001] Docker daemon                                 commit=5604cbe graphdriver=aufs version=1.11.1
INFO[0001] API listen on /var/run/docker.sock     

(type <CTRL>-C)

$> docker daemon -H fd://
FATA[0000] No sockets found. Make sure the docker daemon was started by systemd. 

$> systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

$> systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Qui 2016-05-19 22:03:41 BRT; 22s ago
     Docs: https://docs.docker.com
  Process: 7057 ExecStart=/usr/bin/docker daemon -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 7057 (code=exited, status=1/FAILURE)

Mai 19 22:03:41 thundera systemd[1]: Starting Docker Application Container Engine...
Mai 19 22:03:41 thundera docker[7057]: time="2016-05-19T22:03:41.436568496-03:00" level=fatal 
msg="No sockets found. Make sure the docker daemon was started by systemd."
Mai 19 22:03:41 thundera systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Mai 19 22:03:41 thundera systemd[1]: Failed to start Docker Application Container Engine.
Mai 19 22:03:41 thundera systemd[1]: docker.service: Unit entered failed state.
Mai 19 22:03:41 thundera systemd[1]: docker.service: Failed with result 'exit-code'.

$> journalctl -xe
-- Unit docker.service has begun starting up.
Mai 19 22:03:41 thundera docker[7057]: time="2016-05-19T22:03:41.436568496-03:00" level=fatal 
msg="No sockets found. Make sure the docker daemon was started by systemd."
Mai 19 22:03:41 thundera systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Mai 19 22:03:41 thundera systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed

Describe the results you received: Docker daemon doesn’t run

Describe the results you expected: Docker daemon should be running

Additional information you deem important (e.g. issue happens only occasionally):

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 55 (21 by maintainers)

Commits related to this issue

Most upvoted comments

@gregdurham did you make modifications to the unit file, or using a drop-in file? Docker 1.12 on CentOS no longer uses socket activation, so you need to remove the -H fd:// from the systemd unit file

@ashneo76 you can change that drop-in file, change docker daemon to dockerd and remove -H fd://

@ashneo76 @thaJeztah maybe you can try this

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock

After issuing systemctl unmask docker.socket, I was able to start docker normally (systemctl start docker).

Here’s a suggestion to improve the error message when the socket is masked:

$> systemctl start docker
Unable to start docker.service, since it depends on a masked unit (docker.sock).
Try unmasking it first (systemctl unmask docker.sock).

In addition to that, the process of installing docker (via either get.docker.com or apt-get install docker-engine) should make sure it has installed the docker.sock unit in a non-masked state.

@thaJeztah your comment, “Docker 1.12 on CentOS no longer uses socket activation”, and suggestion on removing -H fd:// worked for me! Thank you very much. Curious though if that bit of wisdom is documented anywhere? I searched the release notes, saw the client daemon split #20639 (another recommendation of yours here), but didn’t see any other good leads to your solution.

dockerd docs has a wealth of info on the systemd config options, but don’t see anything related to your tip apart from the solid socket activation article link which I’m still processing and didn’t see your tip either. I then thought it might be a RedHat change, and my google fu might not be strong today, but didn’t find anything conclusive. I also have seen all the socket removal tickets that were closed with 1.12 GA, but most of the remnant problems are people with drop-ins they forgot about.

I’d love to help out and add a note into any documentation (been wanting to contribute for a while now) if you agree it should be there. Thanks!

@reloxx13 Hmm, perhaps I was too quick to respond.

My docker service was having issues. Google brought me here. @e-user pointed me in the right direction and now everything works again.

  • Ubuntu 18.04
  • Docker version 18.09.0, build 4d60db4
  • I have no idea what I am doing, but I got it working again. I don’t know what -H fd:// does, why it was there, and why it suddenly stopped working.

I am having this same issue on centos 7 “docker-engine-1.12.0-1.el7.centos.x86_64” where it worked properly on “docker-engine-1.11.2-1.el7.centos.x86_64”

Replacing -H fd:// with -H unix:// worked for me. (Don’t forget to reload the service daemon 😃)

Even though this appears to be an old issue, if anyone else comes here they should know that adding -H unix:// will actually allow docker to start because that’s what provides /var/lib/docker.sock and is the default in absence of any -H option.

as outlined above for centos 7.2, removing -H fd:// from /etc/systemd/system/docker.service.d/docker.conf fixed it for me too

Hm, wondering if this has to do with socket activation. What does

systemctl status docker.socket

Give you?

If you’re running on Docker 18.09.0, this is being tracked through https://github.com/docker/for-linux/issues/476 and https://github.com/moby/moby/issues/38119, and will be fixed through https://github.com/docker/docker-ce-packaging/pull/257 as part of the 18.09.1 patch release, which should become available soon.

Here How I solved this issue On centos 7. In /etc/systemd/system/docker.service.d/override.conf ExecStart= ExecStart=/usr/bin/docker daemon --storage-driver=overlay -H fd:// I replaced docker daemon by dockerd and removed -H fd:// then restarted the docker service.

@azhagarasu the -H fd:// is only used for “socket activation” (basically, it allows connecting to /var/run/docker.sock and docker is automatically started if it’s not running), however, if the docker.socket service isn’t running, that doesn’t work. The feature caused issues on RPM based installs, so was removed from the systemd unit files on those systems.

In practice, removing -H fd:// won’t make any difference, unless you were using socket activation

Further digging revealed the following in: /etc/systemd/system/docker.service.d/docker_options.conf

[Service]
EnvironmentFile=-/etc/sysconfig/docker
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// $INSECURE_REGISTRY

Perhaps a left over of an older docker installation.

Hopefully this helps.

@thaJeztah That solved my issue. Thank you

If You upgraded to Docker v1.12.2 from older version and it won’t start service, this helped me: rm /etc/systemd/system/docker.service.d/docker-defaults.conf

@thaJeztah changing docker daemon to dockerd and removing -H fd:// worked. But could you explain what is the function of -H fd:// ?

I created a dropin file to change Docker Root Dir. Before it was like,

ExecStart=/usr/bin/docker daemon -H fd:// --graph="/mnt/block_drive/docker" --storage-driver=devicemapper

Now it is,

ExecStart=/usr/bin/dockerd --graph="/mnt/block_drive/docker" --storage-driver=devicemapper

What is the impact of altering this? Thanks.

I came across the same problem on centos 7.2. Removing the -H fd:// does fix the problem and docker runs fine. Yet, my problem is a bit different, since I’m trying to install DC/OS and the installer keeps changing the file. I’m following this instructions, I followed the official docker instructions, but the DC/OS installerfails every time, complaining about docker. Here’s the output for systemctl status docker

● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Qua 2016-08-10 21:10:20 BST; 2min 47s ago
     Docs: https://docs.docker.com
  Process: 2049 ExecStart=/usr/bin/docker daemon --storage-driver=overlay -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 2049 (code=exited, status=1/FAILURE)

Ago 10 21:10:19 mst01 systemd[1]: Starting Docker Application Container Engine...
Ago 10 21:10:20 mst01 docker[2049]: time="2016-08-10T21:10:20.142801096+01:00" level=fatal msg="no sockets found via socket activation: make sure the service was started by systemd"
Ago 10 21:10:20 mst01 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Ago 10 21:10:20 mst01 systemd[1]: Failed to start Docker Application Container Engine.
Ago 10 21:10:20 mst01 systemd[1]: Unit docker.service entered failed state.
Ago 10 21:10:20 mst01 systemd[1]: docker.service failed.

Output for systemctl status docker.socket

● docker.socket
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

Here’s the result file, after DC/OS changing it back ( adding back the -H fd://)

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon --storage-driver=overlay -H fd://

I’m not sure what to do… any help would be much appreciated. Thanks

I uninstalled docker (apt-get remove docker-engine) and rm -rf /var/lib/dockered before attempting get.docker.com, so it’s a fresh install unless the uninstallation process isn’t removing all it should.

I guess that you could make it delete all systemd unit configuration files that the installation created. This would probably avoid the problem of having a masked unit as a leftover.

I have not checked that, so can you please confirm if all files installed are removed when docker is uninstalled (save for /var/lib/docker perhaps).