moby: hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3)
When trying to build a docker container for a Python data-science environment I’m getting the mentioned error right at the end of the build. Unfortunately it doesn’t appear to happen on less complicated builds.
Error message at end of docker build
re-exec error: exit status 1: output: time="2017-04-26T15:58:20+10:00" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\846463084a548600306c3f05fd185b7b46f04695bae40908f6485d3989b5e291 flavour=1 folder=C:\\Windows\\TEMP\\hcs235647079"
hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\?\C:\ProgramData\docker\windowsfilter\846463084a548600306c3f05fd185b7b46f04695bae40908f6485d3989b5e291 flavour=1 folder=C:\Windows\TEMP\hcs235647079
Output of docker version
:
Client:
Version: 17.06.0-dev
API version: 1.30
Go version: go1.7.5
Git commit: d4fb626
Built: Wed Apr 26 03:46:47 2017
OS/Arch: windows/amd64
Server:
Version: 17.06.0-dev
API version: 1.30 (minimum version 1.24)
Go version: go1.7.5
Git commit: d4fb626
Built: Wed Apr 26 03:46:47 2017
OS/Arch: windows/amd64
Experimental: true
Output of docker info
:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 13
Server Version: 17.06.0-dev
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: l2bridge l2tunnel nat null overlay transparent
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1066.amd64fre.rs1_release_sec.170327-1835)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16GiB
Name: *******
ID: <redacted>
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: <redacted>
Https Proxy: <redacted>
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details: This is on a Windows Server 2016 VM running on vSphere
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 1
- Comments: 81 (30 by maintainers)
Commits related to this issue
- Windows: Pass back system errors on container exit Signed-off-by: John Howard <jhoward@microsoft.com> While debugging #32838, it was found (https://github.com/moby/moby/issues/32838#issuecomment-356... — committed to microsoft/docker by lowenna 6 years ago
- Temporarily disable Windows builds We run into https://github.com/moby/moby/issues/32838 with astonishing regularity, both on ltsc2016 and 1709 based image builds. When this happens, the official im... — committed to infosiftr/mongo by tianon 6 years ago
- Windows: Pass back system errors on container exit Signed-off-by: John Howard <jhoward@microsoft.com> While debugging #32838, it was found (https://github.com/moby/moby/issues/32838#issuecomment-356... — committed to microsoft/docker by lowenna 6 years ago
- Windows: Pass back system errors on container exit Signed-off-by: John Howard <jhoward@microsoft.com> While debugging #32838, it was found (https://github.com/moby/moby/issues/32838#issuecomment-356... — committed to docker/docker-ce by lowenna 6 years ago
…so I’m finding that on anything other than simple builds the
hcsshim::ImportLayer
error happens all the time 😦It would be great to know if anyone else is also seeing it or if it’s something specific to my setup. If the latter there’s some hope I can work around it.
Since I’ve also seen the error on my home laptop running Win10 with no proxy or other corporate issues to muddy the waters, I suspect it’s an actual bug in the docker implementation.
Edit: I spoke too soon on this. I can build in certain circumstances but not others, see https://github.com/Microsoft/mssql-docker/issues/194
I was getting the “ImportLayer failed in Win32” error when trying to build a mssql-server-windows dockerfile. I solved the error by increasing the container size and then rebuilding with --no-cache: https://akosdudas.wordpress.com/2017/12/29/docker-image-build-with-error-importlayer-failed-in-win32-the-system-cannot-find-the-path-specified/
OK, I can repro with the dockerfile from @mback2k above, and with a cut-down smaller repro.
In that dockerfile, some files are being left in the recycle bin which have “odd” filenames. It looks currently like a golang bug being able to interpret Unicode characters in filenames. I just used these files, and wrote a small program which does ioutil.ReadDir to enumerate through files, and call os.Stat on each file. Stat calls GetFileAttributeEx which fails in the same way as debugging docker.
To add another useful hint for layer-related issues like this, the docker-ci-zap tool is just awesome in cleaning up the local cache of Docker image layers that are almost impossible to delete manually: https://github.com/jhowardmsft/docker-ci-zap
For anyone else who ends up here from Google like me, I was trying to create the build tools container as described in the official Microsoft Documentation (followed verbatim). I was encountering this error every time I tried to build the image, but the following steps has my build working consistently:
I’m pretty confident that switching to 1709 did not solve the problem, as I had applied this before the reinstall steps and it did not resolve. It could have been some combination of cached images, etc that were causing this to fail which gets resolved by a reinstall of Docker.
System Info
Docker Info
This appears to have happened to me because of default storage limits, see: https://stackoverflow.com/questions/46539165/docker-windows-build-fails-with-error-the-system-cannot-find-the-path-specifie
Upping the default container base size addressed the issue.
@danechambers Just an FYI, still investigating, but this particular one is a Windows bug, external to Docker. The RS1 utility VM hosting the container is bug-checking during shutdown. It’s not clear why yet, I’ll be handing it off to others internally @ MS to analyze further. It’s also shown a bug in docker which I will submit a fix for - the actual error above is extremely misleading as it should never have tried to commit the layer of a corrupt filesystem due to the shutdown bugcheck earlier. Very occasionally I can also hit another failure which I’ll also investigate if I can repro it again.
For me, a similar error happens when i pull an image on a server 2016 from a registry where the images has been pushed from windows 10 1709:
If i build the exactly image directly on the server 2016 node, it will work. I tried
17.11.0-ce
and17.09.0-ce
on my windows 10 node-> same result.17.06.2-ee-5
is running on my windows server 2016 nodes and both can not pull the image.Thanks.
I’m not sure if it is a permanent solution, but using Windows 10 Insider Preview 17120.1 (rs4_release) and microsoft/windowsservercore-insider I could overcome https://github.com/docker/for-win/issues/1052. (I have tried a few times and all of it worked without any error.)
However adding
--squash
to the build command still produces the same error.Oh I meant I get the same error. Not a new topic. I just didn’t want to paste the redundant error info. I pasted the Dockerfile mainly to illustrate that I am doing nothing fancy/magical/complex. Sorry about that.
Possibly related: https://github.com/moby/moby/issues/31253#issuecomment-314525571
@dtma007 - I am getting similar errors to you, though we are likely doing different things. I have been able to reproduce the issue in my scenario - so I’m hoping that this helps.
Running a Jenkins job on a Windows Server 2016 Datacenter edition Amazon EC2 instance. The job runs a Powershell script that does a docker pull of a Windows image from Account A, a docker tag, and then logs in to Account B and does a docker push. That part works fine. The second part of the script does a docker rmi --force to remove the copy of the image from Docker on the Windows Jenkins host. Even that actually removes the local image, but then the next time I try to run the same process, I get the errors. After hours of trial and error, I was able to get past this and reproduce and fix again twice. Below is the lengthy detail:
First error from event log on Windows Jenkins node: hcsshim::DestroyLayer failed in Win32: Access is denied. (0x5) id=c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea flavour=1
Second error: Error releasing layer sha256:18051c4005b6b29f69cdc8c8afadcedbfcf06de4604f567565dd780e63e9bbc1: hcsshim::DestroyLayer failed in Win32: Access is denied. (0x5) id=c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea flavour=1
Third error: Error trying v2 registry: failed to register layer: re-exec error: exit status 1: output: time=“2017-07-11T00:36:42Z” level=error msg=“hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411” hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411
Fourth error: Attempting next endpoint for pull after error: failed to register layer: re-exec error: exit status 1: output: time=“2017-07-11T00:36:42Z” level=error msg=“hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411” hcsshim::ImportLayer failed in Win32: The system cannot find the file specified. (0x2) layerId=\?\C:\ProgramData\docker\windowsfilter\a40dedbfc02b21485a4ac6aa085d196be59dccc68d57c410b568bc4e44d6ecdd flavour=1 folder=C:\Windows\TEMP\hcs040092411
I then RENAMED the following folder: C:\ProgramData\docker\windowsfilter\c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea
Ran the Powershell script again
Next Information event: Layer sha256:18051c4005b6b29f69cdc8c8afadcedbfcf06de4604f567565dd780e63e9bbc1 cleaned up
Fifth error: Error trying v2 registry: failed to register layer: Cannot create layer with missing parent c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: GetFileAttributesEx C:\ProgramData\docker\windowsfilter\c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: The system cannot find the file specified.
Sixth error: Attempting next endpoint for pull after error: failed to register layer: Cannot create layer with missing parent c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: GetFileAttributesEx C:\ProgramData\docker\windowsfilter\c2090764b798c63438ee74f886235307b969db55eef01c9fae1558d1fe5907ea: The system cannot find the file specified.
Seventh error: Handler for POST /v1.25/images/239641032376.dkr.ecr.us-east-1.amazonaws.com/omnious/windows-container-launcher:0.1.13/tag returned error: no such id: 239641032376.dkr.ecr.us-east-1.amazonaws.com/omnious/windows-container-launcher:0.1.13
Renamed folder back and ran job again…and it was successful!
For now, we have removed the cleanup as this is what appears to have directly caused the issue (docker rmi <image> --force). So somewhere in that command, something isn’t getting cleaned up right. Renaming the directory, re-running, then naming it back seems to trigger a good cleanup. I’m still working on troubleshooting this further, and I’m still a newb, but I wanted to comment here since i’m seeing the same errors and have found a way to trigger & resolve them.
Docker version: Client: Version: 1.12.2-cs2-ws-beta API version: 1.25 Go version: go1.7.1 Git commit: 050b611 Built: Tue Oct 11 02:35:40 2016 OS/Arch: windows/amd64
Server: Version: 1.12.2-cs2-ws-beta API version: 1.25 Go version: go1.7.1 Git commit: 050b611 Built: Tue Oct 11 02:35:40 2016 OS/Arch: windows/amd64
Docker info: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 2 Server Version: 1.12.2-cs2-ws-beta Storage Driver: windowsfilter Windows: Logging Driver: json-file Plugins: Volume: local Network: nat null overlay Swarm: inactive Default Isolation: process Kernel Version: 10.0 14393 (14393.1198.amd64fre.rs1_release_sec.170427-1353) Operating System: Windows Server 2016 Datacenter OSType: windows Architecture: x86_64 CPUs: 2 Total Memory: 8 GiB Name: EC2AMAZ-5AK76IV ID: VZIV:QABT:63VB:KUOW:4FOR:BTTB:QULZ:U7KS:CIZK:QWTU:63SI:65EG Docker Root Dir: C:\ProgramData\docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false