moby: error removing container (1.10, 1.11/master) with AUFS

I managed to reate this issue on 1.13 by doing the following:

1. Updated image string in docker-compose
2. docker-compose pull
3. Ran docker-compose create --force-recreate with images still running

At this point dockerd stop responding to http requests until I restarted the service

With dockerd restarted, I now get the following:

ERROR: Driver aufs failed to remove root filesystem

when trying docker-compose create

Same issue, sudo service docker restart fixed it

$ docker info
Containers: 8
 Running: 0
 Paused: 0
 Stopped: 8
Images: 89
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 187
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-86-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.798 GiB
Name: ahr
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

In case it helps anyone else, because I can’t upgrade Docker, the Kernel, or anything else on CircleCI, I’ve resorted to a simple daemon restart:

sudo service docker restart

Not really announced yet, but we also now have nightly builds in our apt/yum repository, e.g.: https://download.docker.com/linux/ubuntu/dists/xenial/pool/nightly/

@Puneeth-n I already did it, I even restarted my computer, but the container still there.

Do you have some link to instruct me how to change to overlay2?

EDIT: I changed to overlay2, after restart the service the dead container is lost. All my images also, but I can pull it again.

in between i stepped over to overlay2, this made me smile again 😃

Some light at the end of tunnel!

More experimentation yields some clues:

#!/bin/bash

for container in $(docker ps -a --filter status="dead" --format '{{.ID}}' --no-trunc)
do
        echo "Container: $container"
        echo "Mounted PIDs:" $(grep -l "$container" /proc/[0-9]*/mountinfo | cut -f 3 -d / | sort -n)
        echo
done

Given script finds PIDs that are related to Docker container path. Sometimes there are only couple of processes running (escaped from container?) and after killing them it is possible to remove container.

Other times, I see that most of processes (event init, pid 1) are related to container path. For example:

Container: 964f191ee2d16621c790985149faae38a6a86800bec3127b037df138bf46f26f
Mounted PIDs: 1 2 3 5 7 8 9 10 11 12 13 15 16 18 19 20 21 22 23 24 25 26 29 30 31 37 38 39 107 110 123 382 390 411 412 413 414 431 432 452 453 593 1000 1025 2592 2687 2689 2690 2791 2828 2853 2940 3066 3085 3193 3218 3236 3299 3318 3354 3370 3374 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3399 3484 3489 3517 3583 3946 3947 4477 4508 4518 4623 4642 5591 5623 5624 5625 5713 5716 5719 5897 5898 5899 5900 5901 5902 6505 6540 8735 8794 8812 8817 8831 8850 8851 9257 10159 10889 11209 11480 12795 14605 14732 14738 14973 15054 15059 15630 16363 16445 16452 16917 17007 17012 17232 17233 17238 17491 17574 17579 17735 18005 18062 18139 18146 18775 18781 19244 19322 19328 19356 19546 19551 19663 19668 19699 20048 20147 20199 20226 20242 20805 20845 20926 20931 21102 21182 21183 21197 21234 21235 21236 21237 21253 21335 21342 21769 21848 21854 26346 26349 27986 30940

What we see in mountinfo?

# grep 964f191ee2d16621c790985149faae38a6a86800bec3127b037df138bf46f26f /proc/1/mountinfo 
1724 18 0:67 / /run/docker/libcontainerd/964f191ee2d16621c790985149faae38a6a86800bec3127b037df138bf46f26f/rootfs rw,relatime - aufs none rw,si=3fbedc05b5b4a1e1,dio,dirperm1

I had same experience - but what @Puneeth-n suggested, upgrading linux kernel to 3.19.x seems to resolve this issue. No aufs errors since upgrade

I just saw this when trying to start a container using the gcplog logdriver that wasn’t able to launch successfully.

docker run -d --name my-container --log-driver=gcplogs --log-opt gcp-log-cmd=true [...]
38e6a733b02a825dc97208ee2436d31353480bfe31cfa4799cd48203d746fe6e
docker: Error response from daemon: Failed to initialize logging driver: unable to connect or authenticate with Google Cloud Logging: googleapi: Error 403: Google Cloud Logging API has not been used in project 784782548624 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/logging/overview?project=784782548624 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., forbidden.

$ docker rm my-container
Error response from daemon: Driver aufs failed to remove root filesystem fd5e668e8cd14c1a7a2405b26ea3d75bdfd9f25525447b20f6b400eff02a7a23: rename /var/lib/docker/aufs/diff/ae3e5b240821fe702b68544188663ef092bf173b33e495678800c6c3ea498f92 /var/lib/docker/aufs/diff/ae3e5b240821fe702b68544188663ef092bf173b33e495678800c6c3ea498f92-removing: device or resource busy

# docker info
Containers: 9
 Running: 5
 Paused: 0
 Stopped: 4
Images: 382
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 609
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 6.338 GiB
Name: web
ID: IXJA:3H47:WFQC:GZPE:3WTJ:5W4P:LEKY:OCCF:CDXQ:IDV7:O23X:K7HQ
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

I couldn’t find reference to that filesystem or any of those directories when groveling around in /proc or via lsof.