moby: Docker occasionally fails to restart Windows containers, both temporarily and permanently

Description

Despite --restart always, Windows containers do not always start after Docker is restarted.

Steps to reproduce the issue:

The behavior is not consistent, so a restart loop is the best way to reproduce it (that I have found so far).

1. Create some containers using docker run --detach --restart always microsoft/windowsservercore powershell.exe -Command "while (1) { Start-Sleep 30 }"
2. In PowerShell, execute while (1) { restart-service docker ; start-sleep 10 ; docker ps -a }
3. Wait some hours.

Describe the results you received:

I performed a 16 hour overnight run with 6 containers on an Azure DS1v2 VM.

The PowerShell transcript stopped recording docker ps -a output after a while, so my results are only approximate but this is what I got:

• During the first part of the run (visible in transcript) 0.4% of container startups failed.
• During the last part of the run (visible in console) 33% of containers had failed permanently and never started again.
• During the last part of the run (visible in console), for the remaining 66% of containers, 8.8% of startups failed.

Statuses for temporary failures followed the pattern Exited (1073807364) 38 seconds ago or Exited (2147483648) 2 minutes ago.

Statuses for permanent failures followed the pattern Exited (2147483648) 5 hours ago

Describe the results you expected:

Containers start after every Docker restart if they are configured with --restart always

Additional information you deem important (e.g. issue happens only occasionally):

I have also seen exit code 1067 when the container entry point was cmd.exe or ServiceMonitor.exe (using the microsoft/iis image) instead of PowerShell.

The issue occurs more often in “real usage”. I first ran into it when restarting Docker during playing around with Docker configuration changes, when out of my 5-10 restarts, most had a single container fail to start up (out of 2 or 3 I had running). Under artificial usage (the restart loop), the issue happens less often.

Output of docker version:

Client:
 Version:      1.14.0-dev
 API version:  1.26
 Go version:   go1.7.3
 Git commit:   e308f6b
 Built:        Tue Nov 22 18:15:55 2016
 OS/Arch:      windows/amd64

Server:
 Version:             1.14.0-dev
 API version:         1.26
 Minimum API version: 1.24
 Go version:          go1.7.3
 Git commit:          e308f6b
 Built:               Tue Nov 22 18:15:55 2016
 OS/Arch:             windows/amd64
 Experimental:        false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 10
Server Version: 1.14.0-dev
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.479.amd64fre.rs1_release.161110-2025)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 1
Total Memory: 3.5 GiB
Name: temphost02
ID: ASKC:36AM:3TCL:UB2Y:4DDO:5HGG:2GJD:5JHE:SZHK:AP2I:4ANZ:WJZS
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Azure DS1v2 VM. Also tried with 1.13-dev, with no difference in results.