moby: docker stop command hangs after daemon reload and restart - docker 17.03

Description Command docker stop hangs. Container is not stopped. Check also docker ps and hangs as well.

Steps to reproduce the issue:

sudo systemctl reload docker
sudo systemctl reload-or-restart docker
sudo systemctl restart docker
docker stop container

Describe the results you received: No results are given from the command - it hangs.

Describe the results you expected: Container is stopped and the command is successfully finished.

Output of docker version:

Docker version 17.03.0-ce, build 60ccb22

Output of docker info:

Containers: 19
 Running: 18
 Paused: 0
 Stopped: 1
Images: 18
Server Version: 17.03.0-ce
Storage Driver: devicemapper
 Pool Name: docker-253:0-134295373-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.546 GB
 Data Space Total: 107.4 GB
 Data Space Available: 26.09 GB
 Metadata Space Used: 11.89 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.136 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.39 GiB
Name: Vulture-Manager
ID: XPLT:7GXH:4MZW:57AC:VBG6:N3RA:O5EO:ZXXY:DWDQ:GYTE:K6Y4:P32K
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Labels:
 containerslots=25
Experimental: false
Cluster Store: consul://10.42.44.96:8500
Cluster Advertise: 10.0.0.42:2375
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

daemon-data-2017-03-22T071457Z.txt goroutine-stacks-2017-03-22T071457Z.txt strace-docker-stop.txt

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 18 (7 by maintainers)

Most upvoted comments

Thanks, looks stuck here: https://github.com/docker/docker/blob/v17.03.0-ce/daemon/monitor.go#L42

goroutine 721 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc42080004c)
	/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc420800040)
	/usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/docker/docker/daemon.(*Daemon).StateChanged(0xc4203a8200, 0xc4215cc780, 0x40, 0xc4215e1658, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/monitor.go:42 +0x2c6
github.com/docker/docker/libcontainerd.(*container).handleEvent.func1()
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/container_unix.go:217 +0x82
github.com/docker/docker/libcontainerd.(*queue).append.func1(0xc420b19d00, 0x0, 0xc42148ba40, 0xc4215b7080)
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/queue_unix.go:28 +0x30
created by github.com/docker/docker/libcontainerd.(*queue).append
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/queue_unix.go:30 +0x170

@mlaventure It looks like there’s some case where the container I/O streams are not closed.

I think blocked on this select:

goroutine 346 [select, 2 minutes]:
github.com/docker/docker/vendor/github.com/tonistiigi/fifo.(*fifo).Read(0xc420a21440, 0xc420a9c000, 0x8000, 0x8000, 0xc42029ed78, 0x421a99, 0x16f9ec0)
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/vendor/github.com/tonistiigi/fifo/fifo.go:138 +0x2a8
bufio.(*Reader).fill(0xc42023d860)
	/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).WriteTo(0xc42023d860, 0x242a620, 0xc4200ed340, 0xc42029ee58, 0x424c01, 0xc42023d860)
	/usr/local/go/src/bufio/bufio.go:472 +0x273
io.copyBuffer(0x242a620, 0xc4200ed340, 0x24294a0, 0xc42023d860, 0x0, 0x0, 0x0, 0x0, 0xffffffffffffffff, 0xffffffffffffffff)
	/usr/local/go/src/io/io.go:380 +0x37b
io.Copy(0x242a620, 0xc4200ed340, 0x24294a0, 0xc42023d860, 0x1, 0xc42029ef80, 0x4184db)
	/usr/local/go/src/io/io.go:360 +0x68
github.com/docker/docker/pkg/pools.Copy(0x242a620, 0xc4200ed340, 0x7fc676a4e4f8, 0xc420a21440, 0xe, 0x5, 0x18eff99)
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/pools/pools.go:60 +0x7b
github.com/docker/docker/container/stream.(*Config).CopyToPipe.func1.1(0x242a620, 0xc4200ed340, 0x7fc676a4e4f8, 0xc420a21440, 0xc420800040)
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/container/stream/streams.go:119 +0x55
created by github.com/docker/docker/container/stream.(*Config).CopyToPipe.func1
	/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/container/stream/streams.go:123 +0x84

cpuguy83 on Mar 22, 2017