Unable to Remove Containers: driver "windowsfilter" failed to remove root filesystem #36218
Comments
|
Having the same issue. Not running in Azure. Tried the latest build of 17.03 as well and having the same issue. |
|
I have also the same problem on windows server 2016: and here is my docker installation: On windows 10 i have not experienced this problem but on Windows server, i cannot remove the containers even after restarting docker. still, i get access denied error. I have stopped the docker and tried to manually remove the container folder from the windowsfilter folder with Administrator account but it will say: After restarting the Server i was able to remove the containers. but the problem is still there and restarting the server is not an option :) |
|
Any update on this issue? I am also seeing it on Windows Server 2016 build 10.0.14393. |
|
So after lots and lots of messing around I've come to the understanding that it has to be something environmental related that no-one has much of an idea of. I've managed to get Docker fully working on another server but sitting in a different rack that we own. The only different in the servers is the rack it's sitting on. (and the disk type) The initial server that I was experimenting with is in some sort of 'corrupt' state I'm fairly certain. I've removed all the containers, images and pruned all the volumes and I still can't uninstall the Windows 'Containers feature'. It's very important to note here that I performed the installation in an 'offline state'. So I simply downloaded the zip, set up dockerd.exe as a Windows service and then turned it on with a path variable set to docker.exe. So I can't 'uninstall' Docker as a lot of other people are suggesting, you have to manually clean it up yourself. If you can potentially afford it, I'd suggest re-build your Windows Server from scratch and try to get Docker Working before you start putting other things on it. I've tried to contact the Docker EE Support team and have just had my queries ignored/forgotten it seems. |
|
Thanks for the response @Francommit, but unfortunately this is a clean Windows Server 2016 VM that was created specifically for Docker. So there's nothing else that should have interfered with the installation process. We did, however, create a D:\ volume to store Docker data; I'm starting to think this is the root cause of the issue as well. A restart, as you suggested, does actually fix the problem temporarily. After a restart I can delete containers using the CLI. However, if I'm running a Dockerfile (I was following the guide for Windows Containers and Docker 101 at https://www.youtube.com/watch?v=066-9yw8-7c), the intermediate containers created from nanoserver ultimately cause the permissions problem again and I'm back to square one. Below is the output from running the Dockerfile from the tutorial: I would really hope that someone could look at this issue. A scheduled restart to fix this is not a good workaround. I am hoping only creating images is the root cause of the problem but simply don't know at this point. |
|
@coryheuschkel try using the 'default' docker installation (without switching drives) and try to manually import the microsoft/nanoserver container. That's what I've been doing up to this point. On your own machine Then on the server I can't run the very last command and it stops at the HCSSCHIM command: Also raised similar issues here: |
|
Update: After installing Docker, running a container from the I'm very surprised that storing Docker data on a separate volume was not the root cause of the bug. We are currently stuck in the mud with setting up our Windows build server while we wait for some sort of help. Docker version: Docker info: Output from attempting to build |
|
@coryheuschkel I am unable to reproduce the issue following your steps above (at least using a Dockerfile with the first 3 of the 8 steps). Does it reproduce every time? Is your VM local or hosted, and if hosted, where? You mentioned it is clean, so this seems unlikely, but are there any special drivers or scanning software (malware detection, etc.) installed? What does |
|
@johnstep Unfortunately the error comes up every time I try to pull a windowsservercore image. Our VM is local; it's running on a server rack in our offices. We do use Cisco AMP so I suppose it's not truly a "clean" VM but I am working to get the Docker program data directory whitelisted. I'll go ahead and dump the output of |
|
It is possible that ImmunetProtectDriver is scanning the directory and somehow preventing it from being renamed. It also likely slows things down as well. You might be able to remove the contrainers sometime later, but I am not sure. Please let us know what happens with ImmunetProtectDriver disabled, or ignoring the Docker data directory, if possible. |
|
My IT group whitelisted the Docker ProgramData directory on our VM and the issue has not come up again. Cisco AMP (ImmunetProtectDriver) was definitely the culprit here. Thanks for all the help. |
|
So I figured out the problem. (potentially unrelated?) When you tell docker to run in certain modes in windows and it kills the processs it locks all docker functionality. How do I replicate this scenario?
From here all 'docker run' commands fail just before the daemon loads them. The solution? Don't run docker with -it flags where it's not meant to be used. |
|
The -original- issue reported would be fixed by #37712. |
|
Based on containerd/containerd#5422, I suspect this issue may have reappeared (but with a different error message) on Windows Server LTSC 2019 due to changes in go-winio which would have arrived in 0.4.13 (#38541 in 19.03.0) or 0.4.14 (#39679 in 20.10.0). Pending work is to work out if my diagnosis is correct and go-winio needs to be fixed for this API to work on LTSC2019, or if not, what was going on that made it look that way. A possible improvement would be to make the same change as containerd, and use Mainly leaving this comment so this issue can sit on my notifications until the root issue that lead to containerd/containerd#5422 is diagnosed, and to generate relevant cross-issue links. |
|
Same issue. docker : Error response from daemon: container 7a042a0976a9b8db4112075b061d198a14b79128fe166db1c515a7679379b747: driver "windowsfilter" failed to remove root filesystem: failed to detach VHD: failed to open virtual disk: The process cannot access the file because it |
Description
While conducting a load test with Windows containers, Docker commands became unresponsive (expected). In order to recover the system, the Docker service was restarted. After the restart, the previously created containers can't be started or removed.
Steps to reproduce the issue:
Describe the results you received:
Trying to remove the container fails:
Trying to start the container fails:
Describe the results you expected:
I expected after restarting the Docker service for the containers to be stopped (but visible when running "docker ps -a". Running "docker start " should restart the containers. Running "docker rm -f " should remove them.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
The Docker host is running in Azure.
The text was updated successfully, but these errors were encountered: