moby: Unable to Remove Containers: driver "windowsfilter" failed to remove root filesystem
Description
While conducting a load test with Windows containers, Docker commands became unresponsive (expected). In order to recover the system, the Docker service was restarted. After the restart, the previously created containers can’t be started or removed.
Steps to reproduce the issue:
- Add containers until the system becomes unresponsive.
- Restart the Docker service.
- Try to remove one of the containers [FAIL]:
docker rm -f testsite1
- Try to start on of the containers [FAIL].
docker start testsite1
Describe the results you received:
Trying to remove the container fails:
> docker rm -f testsite1
Error response from daemon: driver "windowsfilter" failed to remove root filesystem for 9442da524e13337b7b8b69ad9a26882c94140dfa39460034e2f62f9cf0296837: rename C:\ProgramData\docker\windowsfilter\9442da524e13337b7b8b69ad9a26882c94140dfa39460034e2f62f9cf0296837 C:\ProgramData\docker\windowsfilter\9442da524e13337b7b8b69ad9a26882c94140dfa39460034e2f62f9cf0296837-removing: Access is denied.
Trying to start the container fails:
> docker start testsite1
Error response from daemon: Container is marked for removal and cannot be started.
Error: failed to start containers: testsite1
Describe the results you expected:
I expected after restarting the Docker service for the containers to be stopped (but visible when running “docker ps -a”. Running “docker start <containerName>” should restart the containers. Running “docker rm -f <containerName>” should remove them.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
> docker version
Client:
Version: 17.06.2-ee-6
API version: 1.30
Go version: go1.8.3
Git commit: e75fdb8
Built: Mon Nov 27 22:46:09 2017
OS/Arch: windows/amd64
Server:
Version: 17.06.2-ee-6
API version: 1.30 (minimum version 1.24)
Go version: go1.8.3
Git commit: e75fdb8
Built: Mon Nov 27 22:55:16 2017
OS/Arch: windows/amd64
Experimental: false
Output of docker info:
Containers: 18
Running: 0
Paused: 0
Stopped: 18
Images: 10
Server Version: 17.06.2-ee-6
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: l2bridge l2tunnel nat null overlay transparent
Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 16299 (16299.15.amd64fre.rs3_release.170928-1534)
Operating System: Windows Server Datacenter
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 16GiB
Name: DockerTest
ID: 5IKI:MYU2:AC2W:RABM:W7SX:NPUC:Z2VF:R5BA:43W2:BUM3:V7VR:JSTQ
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Username: harbertc
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
The Docker host is running in Azure.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 15 (4 by maintainers)
Having the same issue. Not running in Azure.
Tried the latest build of 17.03 as well and having the same issue.
I have also the same problem on windows server 2016:
and here is my docker installation:
On windows 10 i have not experienced this problem but on Windows server, i cannot remove the containers even after restarting docker. I also tried to remove it with force like this: docker rm -f -v
still, i get access denied error. I found the other issue also here: https://github.com/moby/moby/issues/30278
I have stopped the docker and tried to manually remove the container folder from the windowsfilter folder with Administrator account but it will say: You need permission to perform this action, you require permission from Administrators to make changes to this folder.
After restarting the Server i was able to remove the containers. but the problem is still there and restarting the server is not an option 😃
It is possible that ImmunetProtectDriver is scanning the directory and somehow preventing it from being renamed. It also likely slows things down as well. You might be able to remove the contrainers sometime later, but I am not sure.
Please let us know what happens with ImmunetProtectDriver disabled, or ignoring the Docker data directory, if possible.
Update: I completely wiped our VM and started fresh with default settings and this issue is present. I am using out of the box configuration for Windows Server 2016 Version 1607 (OS Build 14393.2339) with the current Docker EE version (17.06.2-ee-14) available from installing Docker via Powershell.
After installing Docker, running a container from the
hello-world:nanoserverimage worked OK. However, when running a container from themicrosoft/windowsservercore:ltsc2016image, all of the intermediate containers that are supposed to be removed force the issue to occur.I’m very surprised that storing Docker data on a separate volume was not the root cause of the bug. We are currently stuck in the mud with setting up our Windows build server while we wait for some sort of help.
Docker version:
Docker info:
Output from attempting to build
microsoft/windowsservercore:ltsc2016:@coryheuschkel try using the ‘default’ docker installation (without switching drives) and try to manually import the microsoft/nanoserver container. That’s what I’ve been doing up to this point.
On your own machine
Then on the server
I can’t run the very last command and it stops at the HCSSCHIM command: https://user-images.githubusercontent.com/11533365/41086250-c2c934f0-6a7c-11e8-8c6d-1039ee41b863.png
Also raised similar issues here: https://github.com/docker/for-win/issues/686 https://github.com/moby/moby/issues/34807
Thanks for the response @Francommit, but unfortunately this is a clean Windows Server 2016 VM that was created specifically for Docker. So there’s nothing else that should have interfered with the installation process. We did, however, create a D:\ volume to store Docker data; I’m starting to think this is the root cause of the issue as well.
A restart, as you suggested, does actually fix the problem temporarily. After a restart I can delete containers using the CLI. However, if I’m running a Dockerfile (I was following the guide for Windows Containers and Docker 101 at https://www.youtube.com/watch?v=066-9yw8-7c), the intermediate containers created from nanoserver ultimately cause the permissions problem again and I’m back to square one. Below is the output from running the Dockerfile from the tutorial:
I would really hope that someone could look at this issue. A scheduled restart to fix this is not a good workaround. I am hoping only creating images is the root cause of the problem but simply don’t know at this point.
So after lots and lots of messing around I’ve come to the understanding that it has to be something environmental related that no-one has much of an idea of.
I’ve managed to get Docker fully working on another server but sitting in a different rack that we own. The only different in the servers is the rack it’s sitting on. (and the disk type)
The initial server that I was experimenting with is in some sort of ‘corrupt’ state I’m fairly certain. I’ve removed all the containers, images and pruned all the volumes and I still can’t uninstall the Windows ‘Containers feature’.
It’s very important to note here that I performed the installation in an ‘offline state’. So I simply downloaded the zip, set up dockerd.exe as a Windows service and then turned it on with a path variable set to docker.exe. So I can’t ‘uninstall’ Docker as a lot of other people are suggesting, you have to manually clean it up yourself.
If you can potentially afford it, I’d suggest re-build your Windows Server from scratch and try to get Docker Working before you start putting other things on it.
I’ve tried to contact the Docker EE Support team and have just had my queries ignored/forgotten it seems.