moby: Unexpected Docker Daemon shutdown due to ServeAPI error "The pipe is being closed" when docker stats was used

Description

Docker Daemon stopped when I tried to use following command:

docker stats --no-stream --format 'table{{.BlockIO}}\t{{.Name}}' $(docker ps -qa)

Steps to reproduce the issue:

  1. Restart Docker daemon in debug mode
Stop-Service docker
dockerd -D > daemon.log 2>&1
  1. Pull microsoft/iis image if you do not have it
docker pull microsoft/iis;
  1. In first Powershell console fun script which starts 100 iis containers and then simulate some activity by restarting containers in an infinite loop
for ($i=0; $i -le 100; $i++) {`
	$cname = "test_iis$i"; `
	docker stop $cname; `
	docker rm $cname; `
	$ExtPort = 10000 + $i;`
	docker run -d --name $cname -p $ExtPort`:80 microsoft/iis ping -t localhost; `
}`
$K=0;`
while($True) {` 
	$K++;
	$M=0;`
	$Containers = $(docker ps --format "{{.Names}}");`
	$Count = ($Containers).count;`
	$Containers | foreach { `
		$M++; `
		$Start = Get-Date; echo "$Start`: Restarting #$M/$K (of $Count) $_ ..."; `
		docker restart $_; `
		$Stop = Get-Date; echo "$stop`: #$M (of $Count) $_ started in $(($Stop - $Start).TotalSeconds) s."; `
	}`
}
  1. In second Powershell console run another script which executes docker stats command in an infinite loop
$i=0;`
while($True) {` 
	$i++; echo "$(date): #$i"; docker stats --no-stream --format 'table{{.BlockIO}}\t{{.Name}}' $(docker ps -qa);` 
}
  1. After all 100 containers are started during next 10-20 minutes Docker Daemon will be stopped.

Inside daemon.log I see following information:

time="2017-07-20T19:25:47.500023900Z" level=debug msg="Calling GET /v1.27/containers/cc9d64ef144e/stats?stream=0" 

... (hundreds of stats requests)
time="2017-07-20T19:25:47.572026200Z" level=debug msg="Calling GET /v1.27/containers/2c8d33c286b1/stats?stream=0" 
time="2017-07-20T19:25:47.573027100Z" level=debug msg="Calling GET /v1.27/containers/e558e6949cec/stats?stream=0" 
time="2017-07-20T19:25:47.574026600Z" level=error msg="ServeAPI error: The pipe is being closed." 
time="2017-07-20T19:25:47.578027500Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..." 
...
time="2017-07-20T20:52:36.830675700Z" level=error msg="Force shutdown daemon" 
dockerd : Shutting down due to ServeAPI error: The pipe is being closed.

Interesting thing that if I run container without exposing ports (-p $ExtPort:80) then I can not reproduce this issue quickly. So may be it is related somehow to HNS/networking.

Initially this issue was caught on Docker Daemon 17.03.1-ee-3 and then reproduced by me with 17.06.0-ce

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:30:30 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 22:19:00 2017
 OS/Arch:      windows/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 14
Server Version: 17.06.0-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1358.amd64fre.rs1_release.170602-2252)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16GiB
Name: EC2AMAZ-N2GM0M9
ID: LH3O:5PAX:ATLN:NPF3:THQV:MUQ4:ZIPT:SAYQ:RV53:SG3B:QXGT:6C2I
Docker Root Dir: E:\docker_storage_1_13
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 20
 System Time: 2017-07-20T20:39:05.3530276Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.): It is AWS m4.xlarge instance (1 socket, 4 virtual CPUs, 16 Gib memory)

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

@jorisscheppers The fix was included to 17.06.2-ee-14 (2018-06-21) according to the release notes https://docs.docker.com/ee/engine/release-notes/#runtime-3

I think 18.03.1-ee also contains the fix.

@jbiel I’ve verified we have a tracking issue for potential EE backport of #35968