moby: Cannot stop container with zombie

Description I cannot stop container with zombie process. Docker ps show container is running state.

[root@node0 ~]# docker ps --filter id=25496ab93d95
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                              NAMES
25496ab93d95        safarov/couchdb:2.1.0   "/opt/couchdb/bin/..."   5 hours ago         Up 5 hours          0.0.0.0:5984->5984/tcp, 5986/tcp   couchdb1

But cannot execute any command inside container

[root@node0 ~]# docker exec -it 25496ab93d95 sh
oci runtime error: exec failed: container_linux.go:295: starting container process caused "process_linux.go:86: executing setns process caused \"exit status 21\""

Also top command results

[root@node0 ~]# docker top 25496ab93d95 auxwwf
USER                PID                 %CPU                %MEM                VSZ                 RSS                 TTY                 STAT                START               TIME                COMMAND
root                16964               0.0                 0.0                 0                   0                   ?                   Zl                  Jan01               0:11                | | \_ [beam.smp] <defunct>

Steps to reproduce the issue: Issue definitely have side effect and i not know how to reproduce on other host.

Describe the results you received: I cannot stop/kill container until stop docker service on host.

Describe the results you expected: I expect to kill all container processes and stop container when executed command docker stop

Additional information you deem important (e.g. issue happens only occasionally): none

Output of docker version:

[root@node0 tmp]# docker version
Client:
 Version:      17.10.0-ce
 API version:  1.33
 Go version:   go1.8.3
 Git commit:   f4ffd2511ce93aa9e5eefdf0e912f77543080b0b
 Built:        Fri Nov 17 20:51:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.10.0-ce
 API version:  1.33 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   f4ffd2511ce93aa9e5eefdf0e912f77543080b0b
 Built:        Fri Nov 17 20:52:33 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

[root@node0 tmp]# docker info
Containers: 14
 Running: 11
 Paused: 0
 Stopped: 3
Images: 22
Server Version: 17.10.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: none
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: e7nj67uocrqcqpvybgj8qso8q
 Is Manager: true
 ClusterID: iw12j90oh79yv3ul39okfkcd6
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.31.1.98
 Manager Addresses:
  10.31.1.98:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.13.13-300.fc27.x86_64
Operating System: Fedora 27 (Twenty Seven)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 125.8GiB
Name: node0.docker.rcsnet.ru
ID: UBXK:GAHS:YEWN:BRYI:DRP4:3PXH:A55E:IFXZ:4Z4Z:BTN5:S4NK:7ICQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.): docker started on bare metal server.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 2
  • Comments: 27 (10 by maintainers)

Most upvoted comments

We run into same issue periodically. now with latest v19.03.2 When container gets wedged like this so “docker stop/kill/exec” hangs I just kill -9 the shim pkill -f ContainerID

This cleans it up while not touching other containers on a host

Issue reproduced on 17.12 Docker version

[root@node0 ~]# docker version
Client:
 Version:	17.12.0-ce
 API version:	1.35
 Go version:	go1.9.2
 Git commit:	c97c6d6
 Built:	Wed Dec 27 20:12:17 2017
 OS/Arch:	linux/amd64

Server:
 Engine:
  Version:	17.12.0-ce
  API version:	1.35 (minimum version 1.12)
  Go version:	go1.9.2
  Git commit:	c97c6d6
  Built:	Wed Dec 27 20:14:50 2017
  OS/Arch:	linux/amd64
  Experimental:	false

docker info

[root@node0 ~]# docker info
Containers: 10
 Running: 8
 Paused: 0
 Stopped: 2
Images: 25
Server Version: 17.12.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: none
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: e7nj67uocrqcqpvybgj8qso8q
 Is Manager: true
 ClusterID: iw12j90oh79yv3ul39okfkcd6
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.31.1.98
 Manager Addresses:
  10.31.1.98:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.13.13-300.fc27.x86_64
Operating System: Fedora 27 (Twenty Seven)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 125.8GiB
Name: node0.docker.rcsnet.ru
ID: UBXK:GAHS:YEWN:BRYI:DRP4:3PXH:A55E:IFXZ:4Z4Z:BTN5:S4NK:7ICQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false