moby: Unexpected Docker Daemon shutdown due to ServeAPI error "The pipe is being closed" when docker stats was used
Description
Docker Daemon stopped when I tried to use following command:
docker stats --no-stream --format 'table{{.BlockIO}}\t{{.Name}}' $(docker ps -qa)
Steps to reproduce the issue:
- Restart Docker daemon in debug mode
Stop-Service docker
dockerd -D > daemon.log 2>&1
- Pull microsoft/iis image if you do not have it
docker pull microsoft/iis;
- In first Powershell console fun script which starts 100 iis containers and then simulate some activity by restarting containers in an infinite loop
for ($i=0; $i -le 100; $i++) {`
$cname = "test_iis$i"; `
docker stop $cname; `
docker rm $cname; `
$ExtPort = 10000 + $i;`
docker run -d --name $cname -p $ExtPort`:80 microsoft/iis ping -t localhost; `
}`
$K=0;`
while($True) {`
$K++;
$M=0;`
$Containers = $(docker ps --format "{{.Names}}");`
$Count = ($Containers).count;`
$Containers | foreach { `
$M++; `
$Start = Get-Date; echo "$Start`: Restarting #$M/$K (of $Count) $_ ..."; `
docker restart $_; `
$Stop = Get-Date; echo "$stop`: #$M (of $Count) $_ started in $(($Stop - $Start).TotalSeconds) s."; `
}`
}
- In second Powershell console run another script which executes
docker stats
command in an infinite loop
$i=0;`
while($True) {`
$i++; echo "$(date): #$i"; docker stats --no-stream --format 'table{{.BlockIO}}\t{{.Name}}' $(docker ps -qa);`
}
- After all 100 containers are started during next 10-20 minutes Docker Daemon will be stopped.
Inside daemon.log I see following information:
time="2017-07-20T19:25:47.500023900Z" level=debug msg="Calling GET /v1.27/containers/cc9d64ef144e/stats?stream=0"
... (hundreds of stats requests)
time="2017-07-20T19:25:47.572026200Z" level=debug msg="Calling GET /v1.27/containers/2c8d33c286b1/stats?stream=0"
time="2017-07-20T19:25:47.573027100Z" level=debug msg="Calling GET /v1.27/containers/e558e6949cec/stats?stream=0"
time="2017-07-20T19:25:47.574026600Z" level=error msg="ServeAPI error: The pipe is being closed."
time="2017-07-20T19:25:47.578027500Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
...
time="2017-07-20T20:52:36.830675700Z" level=error msg="Force shutdown daemon"
dockerd : Shutting down due to ServeAPI error: The pipe is being closed.
Interesting thing that if I run container without exposing ports (-p $ExtPort:80
) then I can not reproduce this issue quickly. So may be it is related somehow to HNS/networking.
Initially this issue was caught on Docker Daemon 17.03.1-ee-3
and then reproduced by me with 17.06.0-ce
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version
:
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:30:30 2017
OS/Arch: windows/amd64
Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.24)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 22:19:00 2017
OS/Arch: windows/amd64
Experimental: false
Output of docker info
:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 14
Server Version: 17.06.0-ce
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1358.amd64fre.rs1_release.170602-2252)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16GiB
Name: EC2AMAZ-N2GM0M9
ID: LH3O:5PAX:ATLN:NPF3:THQV:MUQ4:ZIPT:SAYQ:RV53:SG3B:QXGT:6C2I
Docker Root Dir: E:\docker_storage_1_13
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: -1
Goroutines: 20
System Time: 2017-07-20T20:39:05.3530276Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.): It is AWS m4.xlarge instance (1 socket, 4 virtual CPUs, 16 Gib memory)
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (8 by maintainers)
@jorisscheppers The fix was included to 17.06.2-ee-14 (2018-06-21) according to the release notes https://docs.docker.com/ee/engine/release-notes/#runtime-3
I think 18.03.1-ee also contains the fix.
@jbiel I’ve verified we have a tracking issue for potential EE backport of #35968