moby: Docker ps hang

Description

The Docker daemon becomes unresponsive and causes docker ps to hang, containers still continue to run and function fine. A stack trace of the Docker daemon:-

https://gist.github.com/dmyerscough/ced7616a5e8072315e7ea82ef797414c

Steps to reproduce the issue:

Docker daemon runs for 11 hours then the daemon becomes unresponsive

Describe the results you received:

Docker daemon becomes unresponsive.

Describe the results you expected:

Docker daemon shouldn’t become unresponsive.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:02:53 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:02:53 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 19
 Running: 19
 Paused: 0
 Stopped: 0
Images: 78
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null overlay host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 4.4.32
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 125.8 GiB
Name: appdocker242-dca1
ID: 36EB:Y3SF:VHJR:3NIB:IXSX:PDVE:IJ7J:W4KQ:HUOH:V3EQ:FRTX:L6EM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 10.0.0.0/8
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 46 (19 by maintainers)

Most upvoted comments

@mlaventure we also have the problem (we tried with Docker 1.11, 1.12.1 and 1.12.3) that docker ps hangs. Fun fact: we can “always” resolve this by doing strace -p {dockerd-pid} -f.

Do you have a link to the fix/patch in master?

hjacobs on Nov 29, 2016

@hjacobs

Those two PR should do the trick:

mlaventure on Nov 29, 2016

@gservat @tj13 These versions of docker out way past their support period. There are numerous fixes for the docker ps hang symptom. The most recent “stable” release is 17.12 (hot of the presses), or alternatively 17.09 is still supported for another month.

cpuguy83 on Jan 2, 2018

@hjacobs I believe we are pulling these stdio fixes into the 1.12 branch as well.

cpuguy83 on Dec 5, 2016

Hi, I consistently have this issue in my cluster (CoreOS 1235.9.0, Docker 1.12.6) but strace doesn’t seems to help. Both with btrfs and devicemapper storage drivers, on 6 nodes. High CPU usage of dockerd when it happens, reply comes in correlation with normal (<1%) CPU usage.

Should I open another issue?

top output:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                  
 2014 root      20   0 4668452 103732  30812 S 126.2  0.1   9:03.70 dockerd

strace output:

# strace -p 2014   
Process 2014 attached
futex(0x2b06a08, FUTEX_WAIT, 0, NULL^CProcess 2014 detached
 <detached ...>

docker info:

# docker info
Containers: 44
 Running: 20
 Paused: 0
 Stopped: 24
Images: 136
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v4.4.1
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge overlay null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp selinux
Kernel Version: 4.7.3-coreos-r2
Operating System: Container Linux by CoreOS 1235.9.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 125.9 GiB
Name: kube-6
ID: KUKE:NYSF:KEGL:EMV4:AHNU:PPQJ:3HYF:DYJM:I4GS:N454:7FHX:R4CW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Thanks!

mcluseau on Feb 10, 2017