moby: Docker fails to remove containers "driver overlay failed to remove root filesystem: readdirent: no such file or directory"

root@mesos-slave6-qa-uswest2:~# docker rm 4aaff2f0d37a
Error response from daemon: Cannot destroy container 4aaff2f0d37a: Driver overlay failed to remove root filesystem 4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e: readdirent: no such file or directory
FATA[0000] Error: failed to remove one or more containers

root@mesos-slave6-qa-uswest2:~# docker inspect !$
docker inspect 4aaff2f0d37a
[{
    "Config": {
        "Env": [
            "INSTANCE_NO=1",
            "TASK_HOST=mesos-slave6-qa-uswest2.qasql.opentable.com",
            "TASK_REQUEST_ID=ci-umami-bbi-simulator",
            "TASK_DEPLOY_ID=teamcity.2015.06.29T23.52.53",
            "ESTIMATED_INSTANCE_COUNT=1",
            "OT_ENV=ci-uswest2",
            "OT_ENV_WHOLE=ci-uswest2",
            "PORT=31366",
            "PORT0=31366",
            "MESOS_SANDBOX=/mnt/mesos/sandbox",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "APP_DIR=/"
        ],
        "Volumes": null,
        "WorkingDir": ""
    },
    "Created": "2015-07-08T01:54:27.639467476Z",
    "Driver": "overlay",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": [
            "/mnt/mesos-slave/slaves/20150610-165203-2986755594-5050-1666-S8/frameworks/Singularity/executors/ci-umami-bbi-simulator-teamcity.2015.06.29T23.52.53-1436320326525-1-mesos_slave6_qa_uswest2.qasql.opentable.com-us_west_2c/runs/94a87778-21fe-4055-b607-0efe1f08c14a:/mnt/mesos/sandbox"
        ]
    },
    "Name": "/mesos-94a87778-21fe-4055-b607-0efe1f08c14a",
    "State": {
        "Dead": true,
        "Error": "",
        "ExitCode": 0,
        "FinishedAt": "2015-07-08T04:29:51.220130171Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 0,
        "Restarting": false,
        "Running": false,
        "StartedAt": "2015-07-08T01:54:28.883622948Z"
    },
    "Volumes": {
        "/mnt/mesos/sandbox": "/mnt/mesos-slave/slaves/20150610-165203-2986755594-5050-1666-S8/frameworks/Singularity/executors/ci-umami-bbi-simulator-teamcity.2015.06.29T23.52.53-1436320326525-1-mesos_slave6_qa_uswest2.qasql.opentable.com-us_west_2c/runs/94a87778-21fe-4055-b607-0efe1f08c14a"
    },
    "VolumesRW": {
        "/mnt/mesos/sandbox": true
    }
}
]

Indeed, it looks like there has been only partial cleanup of this container:

root@mesos-slave6-qa-uswest2:/mnt/docker/overlay/4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e# ls
merged
root@mesos-slave6-qa-uswest2:/mnt/docker/overlay/4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e# ls merged/
root@mesos-slave6-qa-uswest2:/mnt/docker/overlay/4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e# grep 4aaff2f0d37a /proc/mounts 
overlay /mnt/docker/overlay/4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e/merged overlay rw,relatime,lowerdir=/mnt/docker/overlay/1a347ace77026d6d0d63453adb63b0ca8d130008cf1845166ac1ff7a84d13f50/root,upperdir=/mnt/docker/overlay/4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e/upper,workdir=/mnt/docker/overlay/4aaff2f0d37ac87b8d5df8f41217a438b9418af841405f70b4e138a664cfe00e/work 0 0

root@mesos-slave6-qa-uswest2# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64
root@mesos-slave6-qa-uswest2# docker info
Containers: 132
Images: 692
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Kernel Version: 4.0.4
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 29.45 GiB
Name: mesos-slave6-qa-uswest2
ID: DQ77:L5SM:2ZH6:NHRP:D72D:SIFN:XJDS:7H7Y:MHRR:UX4Y:HTWK:UVB6

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 53 (25 by maintainers)

Commits related to this issue

Most upvoted comments

I am on docker 1.8.3 with overlayfs on extfs. I create and destroy containers at the rate of 1.5/s and I see about 10 Dead containers in the course of an hour.

As a workaround, here is what I run on the box:

#!/usr/bin/env bash

while true
do
  echo Cleaning up dead docker containers...
  docker ps -a -f status=dead -q | xargs -r -I % bash -c 'rm -rf /var/lib/docker/overlay/%1*'
  docker ps -a -f status=dead -q | xargs -r docker rm -f -v
  echo Sleeping 10s before next container cleanup...
  sleep 10
done
+14
tjanczuk on Jan 23, 2016

For anyone encountering this issue on docker for mac, as implied by the error message, my /var/lib/docker folder just did not exist. Solution was simply $ sudo mkdir /var/lib/docker/

+8
JustinLivi on Dec 21, 2016

Another CentOS 7 users.

docker version

Client: Version: 17.03.0-ce API version: 1.26 Go version: go1.7.5 Git commit: 60ccb22 Built: Thu Feb 23 10:54:03 2017 OS/Arch: linux/amd64

Server: Version: 17.03.0-ce API version: 1.26 (minimum version 1.12) Go version: go1.7.5 Git commit: 60ccb22 Built: Thu Feb 23 10:54:03 2017 OS/Arch: linux/amd64 Experimental: false

+4
matejzero on Apr 6, 2017

@dmcgowan Several times in our CI.

+3
ignatov on Jun 21, 2016

Closing since the OP’s issue is fixed by https://github.com/moby/moby/pull/31012/files#diff-723dfe6d49672e6220c7b87d40f7fdd6R24 and is in 17.06.

@yunghoy your issue is different and is addressed by #33960

+1
cpuguy83 on Jul 18, 2017

@soichih having the same issue on CentOS7 too. Did you figure it out? Cheers.

+1
yapartase on Mar 17, 2017

Same here in a Jenkins slave that starts and stops a lot of compositions in parallel on the same machine.

$ docker info
Containers: 74
 Running: 30
 Paused: 0
 Stopped: 44
Images: 1192
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 28.76 GiB
Name: infra-prd-2
ID: X7Z3:5SZP:LXUJ:J2VF:UMUJ:FIEL:2ORL:5HJP:FKG7:RGXD:TKTY:KOJ7
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
+1
pvalsecc on Dec 1, 2016
Read more comments on GitHub
← moby: Docker fails to mount 'd:' drive
moby: Docker fails to run image with error "Invalid Argument" →