moby: hcsshim::PrepareLayer failed in Win32: This operation returned because the timeout period expired. (0x5b4)

Description

Repeatedly receiving “hcsshim::PrepareLayer failed in Win32: This operation returned because the timeout period expired. (0x5b4)” when trying to build images using “docker build” on Server 2016 (docker client/server version 1.12.2-cs2-ws-beta). Rerunning the build (sometimes multiple times) eventually allows the build to complete, but this is annoying. Note that this does not happen every time, but it does occur frequently.

Steps to reproduce the issue:

Build a dockerfile to make a custom image from microsoft/windowsservercore
docker build -t mymyachine .

Describe the results you received:

hcsshim::PrepareLayer failed in Win32: This operation returned because the timeout period expired. (0x5b4) layerId=34268
b729aba561b87438591920c78363a72ed061a53c33da51bb810849df367 flavour=1

Describe the results you expected:

Build should complete succesfully without errors.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens frequently, but not every time. I have been able to successfully build images by running the build over and over again until it succeeds.

Output of docker version:

PS C:\wip\jenkins> docker version
Client:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Server:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Output of docker info:

Containers: 22
 Running: 0
 Paused: 0
 Stopped: 22
Images: 53
Server Version: 1.12.2-cs2-ws-beta
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: nat null overlay
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.321.amd64fre.rs1_release_inmarket.161004-2338)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 6 GiB
Name: POPDOCKER1
ID: KS2Q:ORLC:HZR3:UAI5:YMOB:NYND:JMM6:CSBG:E5YO:GPJ5:4E42:4GWU
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

VMWare VM running Server2016 GA on-prem.

https://raw.githubusercontent.com/Microsoft/Virtualization-Documentation/master/windows-server-container-tools/Debug-ContainerHost/Debug-ContainerHost.ps1 Output

Checking for common problems
Describing Windows Version and Prerequisites
 [+] Is Windows 10 Anniversary Update or Windows Server 2016 1.38s
 [+] Has KB3192366, KB3194496, or later installed if running Windows build 14393 231ms
 [+] Is not a build with blocking issues 42ms
Describing Docker is installed
 [+] A Docker service is installed - 'Docker' or 'com.Docker.Service'  143ms
 [+] Service is running 36ms
 [+] Docker.exe is in path 2.11s
Describing User has permissions to use Docker daemon
 [+] docker.exe should not return access denied 60ms
Describing Windows container settings are correct
 [+] Do not have DisableVSmbOplock set to 1 52ms
 [+] Do not have zz values set 66ms
Describing The right container base images are installed
 [+] At least one of 'microsoft/windowsservercore' or 'microsoft/nanoserver' should be installed 338ms
Showing output from: docker version
Client:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Server:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Showing output from: docker info
Containers: 22
 Running: 0
 Paused: 0
 Stopped: 22
Images: 53
Server Version: 1.12.2-cs2-ws-beta
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: nat null overlay
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.321.amd64fre.rs1_release_inmarket.161004-2338)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 6 GiB
Name: POPDOCKER1
ID: KS2Q:ORLC:HZR3:UAI5:YMOB:NYND:JMM6:CSBG:E5YO:GPJ5:4E42:4GWU
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Warnings & errors from the last 24 hours

   ProviderName: docker

TimeCreated                     Id LevelDisplayName Message

---

10/20/2016 9:21:53 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/20/2016 9:17:16 AM            1 Error            process 7748 in container 1f3d75a838e31d1665e6b2677a38d33433b2ad...
10/20/2016 9:10:26 AM            1 Error            process 6044 in container 74d0590cc6c2734ee51ee1e8949b683019c032...
10/20/2016 9:08:44 AM            1 Error            process 4040 in container 1a3c8fa0800a072f5492e6e91ca59d0d8a095e...
10/20/2016 8:58:23 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/20/2016 8:58:11 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/20/2016 8:58:01 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/20/2016 8:56:02 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/20/2016 8:12:46 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/20/2016 8:05:25 AM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 3:40:51 PM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 2:53:39 PM            1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 2:42:26 PM            1 Error            process 7544 in container 8a20d3b61900357e7925e29b477d98108706eb...
10/19/2016 2:21:24 PM            1 Error            process 6392 in container 0a25b29217978d189097bcc8c4153451c85f4f...
10/19/2016 12:04:23 PM           1 Error            process 5628 in container 7c2b88ece7596f0b505eca2368a3907a6cd947...
10/19/2016 12:03:45 PM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 12:03:37 PM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 12:03:28 PM           1 Error            process 4656 in container 0a680dffed25581a4b9ce91bc22c2d0aa68c75...
10/19/2016 12:03:00 PM           1 Error            Handler for PUT /v1.25/containers/0a680dffed25/archive returned ...
10/19/2016 12:03:00 PM           1 Error            Handler for HEAD /v1.25/containers/0a680dffed25/archive returned...
10/19/2016 12:02:48 PM           1 Error            Handler for PUT /v1.25/containers/0a680dffed25/archive returned ...
10/19/2016 12:02:48 PM           1 Error            Handler for HEAD /v1.25/containers/0a680dffed25/archive returned...
10/19/2016 12:02:32 PM           1 Error            Handler for PUT /v1.25/containers/0a680dffed25/archive returned ...
10/19/2016 12:02:32 PM           1 Error            Handler for HEAD /v1.25/containers/0a680dffed25/archive returned...
10/19/2016 11:58:23 AM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 11:55:51 AM           1 Error            process 7416 in container e69f7b27bc93fd1ad261dd42a5ecd24ef295ee...
10/19/2016 11:52:36 AM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 11:52:28 AM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 11:52:19 AM           1 Error            process 1936 in container dc03e05a62881912a28cd866b2e4fd7e9a98ac...
10/19/2016 11:50:36 AM           1 Error            Handler for POST /v1.25/containers/create returned error: invali...
10/19/2016 11:49:55 AM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 11:41:02 AM           1 Error            Handler for POST /v1.25/containers/05285227ee7b417e1a5f8db2fdf77...
10/19/2016 11:41:02 AM           1 Error            Create container failed with error: container 05285227ee7b417e1a...
10/19/2016 11:40:49 AM           1 Error            Handler for POST /v1.25/containers/90efeb70698d3b4f64343c852b70e...
10/19/2016 11:40:49 AM           1 Error            Create container failed with error: container 90efeb70698d3b4f64...
10/19/2016 11:38:48 AM           1 Error            Handler for POST /v1.25/containers/f00c887df3da781e62143a6f233b5...
10/19/2016 11:38:48 AM           1 Error            Create container failed with error: container f00c887df3da781e62...
10/19/2016 11:38:19 AM           1 Error            Handler for POST /v1.25/containers/aa85962ab4b4252b996cac4818efa...
10/19/2016 11:38:18 AM           1 Error            Create container failed with error: container aa85962ab4b4252b99...
10/19/2016 11:35:00 AM           1 Error            process 1496 in container c4775ab20e633247d10b9afd63a6d7710d4684...
10/19/2016 10:26:08 AM           1 Error            hcsshim::PrepareLayer failed in Win32: This operation returned b...
10/19/2016 10:25:40 AM           1 Error            process 7584 in container 6da43493a2df1076f3bd796a11b8ac6bba8016...
10/19/2016 10:20:49 AM           1 Error            process 4732 in container d0836fdd0aba667f75183edcb040765f9c8905...
10/19/2016 10:08:39 AM           1 Error            Handler for POST /v1.25/build returned error: Unknown instructio...
10/19/2016 10:07:55 AM           1 Error            process 3552 in container ca46f85e4f1f3d415df72af80f9cce3055ea25...
10/19/2016 9:57:56 AM            1 Error            process 4440 in container 14cec561a858b537d7a76222b2a2275b6c5ac0...
10/19/2016 9:48:29 AM            1 Error            process 3216 in container 9013f6d0aaf6ed42951bf077d308ce5663c527...

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 1
Comments: 46 (14 by maintainers)

Most upvoted comments

Please, please fix this! This bug is basically ruining Docker on Windows. A few thoughts on the subject:

This definitely seems to happen a lot more on my Windows 10 laptop (32 gigs of RAM) than Windows Server 2016 (with 8 gigs)
This happens on random steps, it’s never consistent. Building again will usually finish.
The longer the Dockerfile (the more layers), the more likely it is to happen. I have Dockerfiles I’ve had to break apart since I just can’t build them in one go.
If this is a Windows bug due to some sort of timeout, can that error be trapped and Docker retry a few times until it succeeds? Unless Microsoft is aware of the bug and actively working on a fix, I feel Docker should attempt some sort of work around even if hacky.

MikeChristensen on Nov 2, 2018

@rayterrill Thanks for the info. With it, we were able to get a repro of this and think we know the cause. This is a Windows bug, I don’t have an ETA on a fix yet.

darstahl on Oct 25, 2016

@rayterrill sure - feel free to open a new issue

PatrickLang on Oct 25, 2016

@drnybble, I have not seen this issue after I re-created all my Docker hosts using Core version of Windows and to be honest I don’t know any good reason to use full GUI version for that purpose.

Anyway, I think that @kallie-b is right person to say what is status on Microsoft side to investigating this issue?

olljanat on Jul 25, 2017

Any idea what causes this issue? Slow disk? I’ve started to get this a lot in the last week or so for seemingly no reason. What kinds of things can I do to mitigate the cause of the issue?

Edit I get this while starting containers

charlessolar on Jan 13, 2017