moby: Container hangs during normal use

Description

Container hangs and is totally uncontactable. This occurs during “normal use”, while I am running a test suite against an image of the product I work on. There is not one specific point or event which causes the hang (I’ve checked using trace) but during a 2-minute long test suite we hit this problem about 90% of times.

Describe the results you received: Docker container hangs. None of docker kill [container], docker exec [container] bash, docker stop [container], docker logs [container] have any effect nor return. docker ps shows the container, with correct created time, and status = up ‘time since creation’. Restarting the VM running docker does remove the container.

Describe the results you expected: We would like the container not to hang, and we would like the container still to be possible to kill if it must hang.

Additional information you deem important (e.g. issue happens only occasionally): I have tried using sigkill to kill the container, and the container daemon using SIGHUP, -KILL and -USR1, none have any effect.

Output of docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

Containers: 11
 Running: 1
 Paused: 0
 Stopped: 10
Images: 5
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.3.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.701GiB
Name: pc5470-new-dev-env.ad.datcon.co.uk
ID: GYBZ:K5RS:MIF4:IZD4:PCK6:OABY:T2GV:FSQX:DRHZ:6OIV:QCKM:HNBN
Docker Root Dir: /home/.docker_data
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 pesdocker:5000
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.): Virtualbox Version 5.2.12 running CentOS Linux release 7.5.1804 (Core)

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 6
  • Comments: 34 (14 by maintainers)

Most upvoted comments

I have this problem too. Was it introduced with 18.03?

Same problem here

The stack trace above was from 18.06 (which was later than @tommflynn - I hadn’t spotted that he wasn’t on the latest version). Since my setup seems to differ from his, I’ve added my docker version and docker info below. My VirtualBox and OS versions are the same as his

docker version

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:25:29 2018
  OS/Arch:          linux/amd64
  Experimental:     false

docker info

Containers: 5
 Running: 1
 Paused: 0
 Stopped: 4
Images: 6
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.11.6.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 10.34GiB
Name: <removed>
ID: DCLD:WJTC:I7RR:QMZ3:DGOQ:MCZ6:OG5N:JB7C:FHT2:3INV:HBKE:5X34
Docker Root Dir: /home/.docker_data
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 pesdocker:5000
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Same problem here

I’m a colleague of @tommflynn. Here’s the stack trace. Thanks for your help! stack_trace.log

same problem here