moby: docker: Error response from daemon: Container command '' not found or does not exist.. ALWAYS

Output of docker version:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 6
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.4.0-25-lowlatency
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.516 GiB
Name: labs-064
ID: AUXF:FOAH:TAZH:KOJF:CABW:MC4F:UCNT:GZCA:IUSN:5BFF:UHW2:ZD6E
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 11
 Goroutines: 31
 System Time: 2016-06-15T16:05:43.202890723+02:00
 EventsListeners: 0
Username: leandrolucarellasociomantic
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical machine.

Steps to reproduce the issue:

1. docker pull hello-world
2. docker run hello-world

Describe the results you received:

For docker pull hello-world:

Using default tag: latest
latest: Pulling from library/hello-world
a9d36faac0fe: Pull complete 
Digest: sha256:e52be8ffeeb1f374f440893189cd32f44cb166650e7ab185fa7735b7dc48d619
Status: Downloaded newer image for hello-world:latest

OK

For docker run hello-world: docker: Error response from daemon: Container command '/hello' not found or does not exist..

Describe the results you expected:

Same for docker pull hello-world, for docker run hello-world:

Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Additional information you deem important (e.g. issue happens only occasionally):

I know this looks like a duplicate, but I couldn’t find any issues where this happens with every run and every image/container, this is why I’m opening a new issue (I hope that’s fine).

I have a very similar setup in a server and is working fine, because of this I suspect this must be some environment issue, but I’m out of ideas about how to debug this. I’m trying with the same kernel (4.2.0), linux-image-extra and docker-engine package and it works in the server, but it fails in my workstation.

I’m using Ubuntu trusty (as stated in docker info), I’ve tried with the kernels from wily and xenial too. In total I tried with kernels 3.13.0, 4.2.0 and 4.4.0 (both generic and lowlatency flavours).

Some more info about my system:

$ cat /etc/apt/sources.list.d/docker.list
deb https://apt.dockerproject.org/repo ubuntu-trusty main
$ which docker
/usr/bin/docker
$ dpkg -S /usr/bin/docker
docker-engine: /usr/bin/docker
$ debsums -c docker-engine && echo OK
OK

Debug mode:

$ docker -D run hello-world
DEBU[0000] Corrupted prefix: []                         
DEBU[0000] [hijack] End of stdout                       
docker: Error response from daemon: Container command '/hello' not found or does not exist..

Output for the server running in debug mode:

time="2016-06-15T16:20:37.813025132+02:00" level=debug msg="Calling POST /v1.23/containers/create" 
time="2016-06-15T16:20:37.813280882+02:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":false,\"Binds\":null,\"BlkioBps\":0,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioIOps\":0,\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SandboxSize\":0,\"SecurityOpt\":null,\"ShmSize\":0,\"StorageOpt\":null,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"hello-world\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}" 
time="2016-06-15T16:20:37.827207229+02:00" level=debug msg="container mounted via layerStore: /var/lib/docker/aufs/mnt/13f60f63cfc44e7ca150a198b91f7685108fefecbe567b410993f7519a918f4b" 
time="2016-06-15T16:20:37.842396562+02:00" level=debug msg="Calling POST /v1.23/containers/2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f/attach?stderr=1&stdout=1&stream=1" 
time="2016-06-15T16:20:37.842545293+02:00" level=debug msg="attach: stdout: begin" 
time="2016-06-15T16:20:37.842641049+02:00" level=debug msg="attach: stderr: begin" 
time="2016-06-15T16:20:37.842972015+02:00" level=debug msg="Calling POST /v1.23/containers/2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f/start" 
time="2016-06-15T16:20:37.843695397+02:00" level=debug msg="container mounted via layerStore: /var/lib/docker/aufs/mnt/13f60f63cfc44e7ca150a198b91f7685108fefecbe567b410993f7519a918f4b" 
time="2016-06-15T16:20:37.843822449+02:00" level=debug msg="Assigning addresses for endpoint serene_cori's interface on network bridge" 
time="2016-06-15T16:20:37.843849703+02:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])" 
time="2016-06-15T16:20:37.847324182+02:00" level=debug msg="Assigning addresses for endpoint serene_cori's interface on network bridge" 
time="2016-06-15T16:20:37.852984685+02:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers : [nameserver 8.8.8.8 nameserver 8.8.4.4]" 
time="2016-06-15T16:20:37.853015681+02:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers : [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]" 
time="2016-06-15T16:20:37.858754527+02:00" level=debug msg="Programming external connectivity on endpoint serene_cori (ab15f00dceb8c119c419bf6b15831c51c7ddd5000f1105c70f1b8b651fcfb151)" 
time="2016-06-15T16:20:37.913721021+02:00" level=debug msg="sandbox set key processing took 16.679925ms for container 2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f" 
ERRO[1254] containerd: start container                   error=oci runtime error: could not synchronise with container process: no such file or directory id=2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f
time="2016-06-15T16:20:37.945491775+02:00" level=debug msg="attach: stdout: end" 
time="2016-06-15T16:20:37.945671399+02:00" level=debug msg="attach: stderr: end" 
time="2016-06-15T16:20:37.945895913+02:00" level=debug msg="Revoking external connectivity on endpoint serene_cori (ab15f00dceb8c119c419bf6b15831c51c7ddd5000f1105c70f1b8b651fcfb151)" 
time="2016-06-15T16:20:37.985379769+02:00" level=debug msg="Releasing addresses for endpoint serene_cori's interface on network bridge" 
time="2016-06-15T16:20:37.985440881+02:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)" 
time="2016-06-15T16:20:38.008633099+02:00" level=error msg="Handler for POST /v1.23/containers/2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f/start returned error: Container command '/hello' not found or does not exist." 

Files in /var/lib/docker:

# find /var/lib/docker -name \*2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f\* | xargs ls -l
/var/lib/docker/containers/2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f:
total 28
-rw-r--r-- 1 root root 2290 Jun 15 16:20 config.v2.json
-rw-r--r-- 1 root root 1053 Jun 15 16:20 hostconfig.json
-rw-r--r-- 1 root root   13 Jun 15 16:20 hostname
-rw-r--r-- 1 root root  174 Jun 15 16:20 hosts
-rw-r--r-- 1 root root  213 Jun 15 16:20 resolv.conf
-rw------- 1 root root   71 Jun 15 16:20 resolv.conf.hash
drwx------ 2 root root 4096 Jun 15 16:20 shm

/var/lib/docker/image/aufs/layerdb/mounts/2a0dff9fd0d1b27998f6a4d90267307ec28ab2d27a1c26504601a12bfe329e4f:
total 12
-rw-r--r-- 1 root root 69 Jun 15 16:20 init-id
-rw-r--r-- 1 root root 64 Jun 15 16:20 mount-id
-rw-r--r-- 1 root root 71 Jun 15 16:20 parent

I tried reinstalling the package, by completely removing /var/lib/docker and same results.

I also tried with plenty of other images, and running custom commands (like in an ubuntu image running docker run ubuntu /bin/sh or ls or ps, etc. Always the same results, docker: Error response from daemon: Container command '<command>' not found or does not exist.. (where <command> is the command I used.

Evidently the error message is wrong, because I’m pretty sure the /hello command is there in the container (or at least the originating image). (and a very minor detail, there is an extra final dot in the error message 😃.

I would appreciate at least some help on moving forward with ways to continue debugging this, because I can’t get my installation to work and I’m out of ideas on how to get a better error message that tells me what is really actually failing…