moby: hang on `docker stop`

When we stop our Discourse Docker container with docker stop app or service docker restart, it hangs. One or more runsv processes inside the container hang, consuming 100% CPU. top shows the CPU usage as “system” usage, so they are hung inside the kernel somewhere. Because of this, they can’t be killed even with SIGKILL; we need to reboot the server.

This is a regression that showed up after we did some routine updates on the server (the ones that Ubuntu nags you to do). The packages that were upgraded included the kernel and Docker.

root@mail:~# uname -a
Linux mail.daystar.io 3.13.0-73-generic #116-Ubuntu SMP Fri Dec 4 15:31:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@mail:~# docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64
root@mail:~# docker info
Containers: 34
Images: 131
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 230
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-73-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 12
Total Memory: 51.08 GiB
Name: mail.daystar.io
ID: 5PHA:QONL:U6TH:2IPO:AVNK:LMCW:R6XT:AVZZ:KNGC:T4TF:SX5S:5DHN
WARNING: No swap limit support

About this issue

Original URL
State: closed
Created 9 years ago
Reactions: 4
Comments: 21 (8 by maintainers)

Commits related to this issue

docker containers should be stopped before being removed - refs https://github.com/docker/docker/issues/18758 — committed to owncloud/core by DeepDiver1975 8 years ago

Most upvoted comments

@thaJeztah @jfrazelle having just worked around this issue I think docker can (and should) workaround this problem.

The gist of what happens is:

You have a container with N processes
docker stop is issued
pid 1 gets a TERM
pid 1 exists but other pids are still running some may have got TERMs
In some weird kernel linux container condition the orphan processes are not torn down and trigger a failure mode in the kernel.

Instead I thing the “semantics” of docker stop should be changed to:

Send TERM to pid 1, wait up to 10 seconds for pid 1 to finish. (if not finished send a QUIT)
Check if any other pids are still running in the container, if there are a. In parallel send TERM + wait for up to 5 seconds for each of the remaining pids. If it is still running send a QUIT and wait.
Stop container

This will change the semantics a bit on docker stop, but, on the other hand, introduce some sort of documented behavior for orphans and make stop much more reliable even if kernel or aufs has bugs.

+11

SamSaffron on Jan 27, 2016