moby: docker 1.4 and "Driver aufs failed to remove root filesystem", "device or resource busy"

I hit this on docker version: 1.6.0.

I was able to remove the container with docker rm -f

Faced similar issue in deleting an old docker images directory , not in use by new docker-machine installation.

$ sudo rm -rf  docker_images.old/
rm: cannot remove ‘docker_images.old/aufs’: Device or resource busy 

Solved by :

$ sudo service docker stop 
$ sudo modprobe -r aufs  
$ sudo rm -rf  docker_images.old/

I got a similar issue with docker 1.8.0-dev.

Below some useful info:

$ docker info
Containers: 0
Images: 18
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 18
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-55-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 490 MiB
Name: consul-server
ID: 22WL:EEPB:YJIQ:LA4V:UHUO:PTZ3:CQYO:F7ED:3RMS:QV7O:LURE:J4W5
WARNING: No swap limit support
Experimental: true

$ docker version
Client:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   a2ea8f2
 Built:        Sat Jul 25 22:03:05 UTC 2015
 OS/Arch:      linux/amd64
 Experimental: true

Server:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   a2ea8f2
 Built:        Sat Jul 25 22:03:05 UTC 2015
 OS/Arch:      linux/amd64
 Experimental: true

And here the description of my issue:

I had a running consul container. It was started with this command:

docker run -d \
           -p 8500:8500 \
           -p 8300-8302:8300-8302/tcp \
           -p 8300-8302:8300-8302/udp \
           -h consul progrium/consul -server -bootstrap

I tried to remove it with:

$ docker rm -f 5515181938cf

And:

$ docker stop 5515181938cf && docker rm -f 5515181938cf

And:

$ docker kill 5515181938cf && docker rm -f 5515181938cf

But got the same issue:

Error response from daemon: Cannot destroy container 5515181938cf: Driver aufs failed to remove root filesystem 5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076: rename /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076 /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076-removing: device or resource busy
Error: failed to remove containers: [5515181938cf]

Then I restarted the docker service:

$ service docker restart

After that the container was in the Exited state:

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                       PORTS                                                                                                                  NAMES
5515181938cf        progrium/consul     "/bin/start -server -"   About an hour ago   Exited (137) 8 seconds ago   53/tcp, 0.0.0.0:8300-8302->8300-8302/tcp, 8400/tcp, 53/udp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8300-8302->8300-8302/udp   elegant_fermat

But I still couldn’t remove it:

$ docker rm 5515181938cf
Error response from daemon: Cannot destroy container 5515181938cf: Driver aufs failed to remove root filesystem 5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076: rename /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076 /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076-removing: device or resource busy
Error: failed to remove containers: [5515181938cf]

Here is the content of this directory:

$ ls -al /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
total 20
drwxr-xr-x  5 root root 4096 Jul 29 09:14 .
drwxr-xr-x 22 root root 4096 Jul 29 09:14 ..
drwxr-xr-x  2 root root 4096 Jul 29 09:14 data
-r--r--r--  1 root root    0 Jul 29 09:14 .wh..wh.aufs
drwx------  2 root root 4096 Jul 29 09:14 .wh..wh.orph
drwx------  2 root root 4096 Jul 29 09:14 .wh..wh.plnk

lsof returned nothing:

$ lsof +D /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076

I also couldn’t remove it manually:

$ rm -rf /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
rm: cannot remove ‘/var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076’: Device or resource busy

Then I found out that the consul process was still running:

$ pa aux | grep consul
root     30879  0.1  2.0 33960432 10160 ?      Ssl  09:14   0:09 /bin/consul agent -config-dir=/config -server -bootstrap

After killing the process manually I could remove the directory and the container

$ service docker stop
$ kill 30879
$ rm -rf /var/lib/docker/aufs/diff/5515181938cf937a979ffd8feefe3cb1e44793840bb59f87be48cf980c9a3076
$ service docker start
docker start/running, process 31775
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                                                  NAMES
5515181938cf        progrium/consul     "/bin/start -server -"   About an hour ago   Dead                53/tcp, 0.0.0.0:8300-8302->8300-8302/tcp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8300-8302->8300-8302/udp, 53/udp, 8400/tcp   nostalgic_darwin
$ docker rm -f 5515181938cf
5515181938cf
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES