moby: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)

…so I’m finding that on anything other than simple builds the hcsshim::ImportLayer error happens all the time 😦

It would be great to know if anyone else is also seeing it or if it’s something specific to my setup. If the latter there’s some hope I can work around it.

Since I’ve also seen the error on my home laptop running Win10 with no proxy or other corporate issues to muddy the waters, I suspect it’s an actual bug in the docker implementation.

Edit: I spoke too soon on this. I can build in certain circumstances but not others, see https://github.com/Microsoft/mssql-docker/issues/194

I was getting the “ImportLayer failed in Win32” error when trying to build a mssql-server-windows dockerfile. I solved the error by increasing the container size and then rebuilding with --no-cache: https://akosdudas.wordpress.com/2017/12/29/docker-image-build-with-error-importlayer-failed-in-win32-the-system-cannot-find-the-path-specified/

OK, I can repro with the dockerfile from @mback2k above, and with a cut-down smaller repro.

In that dockerfile, some files are being left in the recycle bin which have “odd” filenames. It looks currently like a golang bug being able to interpret Unicode characters in filenames. I just used these files, and wrote a small program which does ioutil.ReadDir to enumerate through files, and call os.Stat on each file. Stat calls GetFileAttributeEx which fails in the same way as debugging docker.

To add another useful hint for layer-related issues like this, the docker-ci-zap tool is just awesome in cleaning up the local cache of Docker image layers that are almost impossible to delete manually: https://github.com/jhowardmsft/docker-ci-zap

For anyone else who ends up here from Google like me, I was trying to create the build tools container as described in the official Microsoft Documentation (followed verbatim). I was encountering this error every time I tried to build the image, but the following steps has my build working consistently:

1. Disable “Container” and “Hyper-V” from Windows Features.
2. Uninstall Docker
3. Reboot
4. Re-enable “Container” and “Hyper-V”. Restart.
5. Download and install latest Docker.
6. Switch to using the 1709 tag of the windows server container.

- FROM microsoft/windowsservercore:1709
+ FROM microsoft/windowsservercore:latest

I’m pretty confident that switching to 1709 did not solve the problem, as I had applied this before the reinstall steps and it did not resolve. It could have been some combination of cached images, etc that were causing this to fail which gets resolved by a reinstall of Docker.

C:\Users\jclay>systeminfo

OS Name:                   Microsoft Windows 10 Pro
OS Version:                10.0.16299 N/A Build 16299
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Workstation
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                        [01]: Intel64 Family 6 Model 158 Stepping 9 GenuineIntel ~4200 Mhz

Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume4
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-05:00) Eastern Time (US & Canada)
Total Physical Memory:     65,436 MB
Available Physical Memory: 47,527 MB
Virtual Memory: Max Size:  75,164 MB
Virtual Memory: Available: 53,605 MB
Virtual Memory: In Use:    21,559 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    WORKGROUP
Logon Server:              \\DESKTOP-CBGV3SI
Hotfix(s):                 14 Hotfix(s) Installed.
                        [01]: KB4048951
                        [02]: KB4049179
                        [03]: KB4053577
                        [04]: KB4054022
                        [05]: KB4056887
                        [06]: KB4058702
                        [07]: KB4074595
                        [08]: KB4074608
                        [09]: KB4087256
                        [10]: KB4088785
                        [11]: KB4090914
                        [12]: KB4093110
                        [13]: KB4099989
                        [14]: KB4093112

Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 8
Server Version: 18.03.0-ce
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 16299 (16299.15.amd64fre.rs3_release.170928-1534)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 63.9GiB
Name: DESKTOP-CBGV3SI
ID: YDNP:SRO4:M7J7:WRQV:T6FP:5FJK:WWQQ:WOCV:OHIN:PVXK:H2AM:5Y3H
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: -1
Goroutines: 37
System Time: 2018-04-12T15:49:45.7566067-04:00
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

This appears to have happened to me because of default storage limits, see: https://stackoverflow.com/questions/46539165/docker-windows-build-fails-with-error-the-system-cannot-find-the-path-specifie

Upping the default container base size addressed the issue.

@danechambers Just an FYI, still investigating, but this particular one is a Windows bug, external to Docker. The RS1 utility VM hosting the container is bug-checking during shutdown. It’s not clear why yet, I’ll be handing it off to others internally @ MS to analyze further. It’s also shown a bug in docker which I will submit a fix for - the actual error above is extremely misleading as it should never have tried to commit the layer of a corrupt filesystem due to the shutdown bugcheck earlier. Very occasionally I can also hit another failure which I’ll also investigate if I can repro it again.

For me, a similar error happens when i pull an image on a server 2016 from a registry where the images has been pushed from windows 10 1709:

failed to register layer: re-exec error: exit status 1: output: time="2017-11-27T09:31:11+01:00" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\8f9191263fd2a5ae93c29ba554426c4adce8fcf333c1b9bf4f217556650d628e flavour=1 folder=C:\\Windows\\TEMP\\hcs456696507"
hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\?\C:\ProgramData\docker\windowsfilter\8f9191263fd2a5ae93c29ba554426c4adce8fcf333c1b9bf4f217556650d628e flavour=1 folder=C:\Windows\TEMP\hcs456696507

If i build the exactly image directly on the server 2016 node, it will work. I tried 17.11.0-ce and 17.09.0-ce on my windows 10 node-> same result. 17.06.2-ee-5 is running on my windows server 2016 nodes and both can not pull the image.

Thanks.

I’m not sure if it is a permanent solution, but using Windows 10 Insider Preview 17120.1 (rs4_release) and microsoft/windowsservercore-insider I could overcome https://github.com/docker/for-win/issues/1052. (I have tried a few times and all of it worked without any error.)

However adding --squash to the build command still produces the same error.

Oh I meant I get the same error. Not a new topic. I just didn’t want to paste the redundant error info. I pasted the Dockerfile mainly to illustrate that I am doing nothing fancy/magical/complex. Sorry about that.

Possibly related: https://github.com/moby/moby/issues/31253#issuecomment-314525571

@dtma007 - I am getting similar errors to you, though we are likely doing different things. I have been able to reproduce the issue in my scenario - so I’m hoping that this helps.

Running a Jenkins job on a Windows Server 2016 Datacenter edition Amazon EC2 instance. The job runs a Powershell script that does a docker pull of a Windows image from Account A, a docker tag, and then logs in to Account B and does a docker push. That part works fine. The second part of the script does a docker rmi --force to remove the copy of the image from Docker on the Windows Jenkins host. Even that actually removes the local image, but then the next time I try to run the same process, I get the errors. After hours of trial and error, I was able to get past this and reproduce and fix again twice. Below is the lengthy detail:

First error from event log on Windows Jenkins node: hcsshim::DestroyLayer failed in Win32: Access is denied. (0x5) id=c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea flavour=1

Second error: Error releasing layer sha256:18051c4005b6b29f69cdc8c8afadcedbfcf06de4604f567565dd780e63e9bbc1: hcsshim::DestroyLayer failed in Win32: Access is denied. (0x5) id=c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea flavour=1

Third error: Error trying v2 registry: failed to register layer: re-exec error: exit status 1: output: time=“2017-07-11T00:36:42Z” level=error msg=“hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411” hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411

Fourth error: Attempting next endpoint for pull after error: failed to register layer: re-exec error: exit status 1: output: time=“2017-07-11T00:36:42Z” level=error msg=“hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411” hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411

I then RENAMED the following folder: C:\ProgramData\docker\windowsfilter\c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea

Ran the Powershell script again

Next Information event: Layer sha256:18051c4005b6b29f69cdc8c8afadcedbfcf06de4604f567565dd780e63e9bbc1 cleaned up

Fifth error: Error trying v2 registry: failed to register layer: Cannot create layer with missing parent c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: GetFileAttributesEx C:\ProgramData\docker\windowsfilter\c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: The system cannot find the file specified.

Sixth error: Attempting next endpoint for pull after error: failed to register layer: Cannot create layer with missing parent c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: GetFileAttributesEx C:\ProgramData\docker\windowsfilter\c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: The system cannot find the file specified.

Seventh error: Handler for POST /v1.25/images/239641032376.dkr.ecr.us-east-1.amazonaws.com/omnious/windows-container-launcher:0.1.13/tag returned error: no such id: 239641032376.dkr.ecr.us-east-1.amazonaws.com/omnious/windows-container-launcher:0.1.13

Renamed folder back and ran job again…and it was successful!

For now, we have removed the cleanup as this is what appears to have directly caused the issue (docker rmi <image> --force). So somewhere in that command, something isn’t getting cleaned up right. Renaming the directory, re-running, then naming it back seems to trigger a good cleanup. I’m still working on troubleshooting this further, and I’m still a newb, but I wanted to comment here since i’m seeing the same errors and have found a way to trigger & resolve them.

Docker version: Client: Version: 1.12.2-cs2-ws-beta API version: 1.25 Go version: go1.7.1 Git commit: 050b611 Built: Tue Oct 11 02:35:40 2016 OS/Arch: windows/amd64

Server: Version: 1.12.2-cs2-ws-beta API version: 1.25 Go version: go1.7.1 Git commit: 050b611 Built: Tue Oct 11 02:35:40 2016 OS/Arch: windows/amd64

Docker info: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 1.12.2-cs2-ws-beta Storage Driver: windowsfilter Windows: Logging Driver: json-file Plugins: Volume: local Network: nat null overlay Swarm: inactive Default Isolation: process Kernel Version: 10.0 14393 (14393.1198.amd64fre.rs1_release_sec.170427-1353) Operating System: Windows Server 2016 Datacenter OSType: windows Architecture: x86_64 CPUs: 2 Total Memory: 8 GiB Name: EC2AMAZ-5AK76IV ID: VZIV:QABT:63VB:KUOW:4FOR:BTTB:QULZ:U7KS:CIZK:QWTU:63SI:65EG Docker Root Dir: C:\ProgramData\docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false