moby: After stopping docker, previously running containers cannot be started or removed
The issue can be reproduced as follows:
$ docker run -d ubuntu:trusty tail -f /dev/null
c39206003c7ae8992a554a9ac2ea130327fc4af1b2c389656c34baf9a56c84b5
$ stop docker
docker stop/waiting
$ start docker
docker start/running, process 2389
$ docker ps -q
# prints nothing...
$ docker ps -a -q
c39206003c7a
$ docker start c39206003c7a
Error: Cannot start container c39206003c7a: Error getting container c39206003c7ae8992a554a9ac2ea130327fc4af1b2c389656c34baf9a56c84b5 from driver devicemapper: Error mounting '/dev/mapper/docker-253:0-267081-c39206003c7ae8992a554a9ac2ea130327fc4af1b2c389656c34baf9a56c84b5' on '/var/lib/docker/devicemapper/mnt/c39206003c7ae8992a554a9ac2ea130327fc4af1b2c389656c34baf9a56c84b5': device or resource busy
2014/05/08 19:14:57 Error: failed to start one or more containers
$ docker rm c39206003c7a
Error: Cannot destroy container c39206003c7a: Driver devicemapper failed to remove root filesystem c39206003c7ae8992a554a9ac2ea130327fc4af1b2c389656c34baf9a56c84b5: Error running removeDevice
2014/05/08 19:15:15 Error: failed to remove one or more containers
This is an up to date Ubuntu 14.04 host running lxc-docker 0.11.1. Storage driver is devicemapper and kernel version is 3.13.0.
This is a regression from docker 0.9 (from the official Ubuntu repos). The problem is also present in 0.10.
About this issue
- Original URL
- State: closed
- Created 10 years ago
- Reactions: 20
- Comments: 113 (23 by maintainers)
Commits related to this issue
- Força a parada de todos os serviços durante a atualização. Durante o processo de atualização, estamos acertando um erro[1] do Docker quando ele tenta lidar com alguns processos. Isso ocorre quando o... — committed to bltavares/docker by bltavares 8 years ago
- docker/docker#5684(unrestartable containers) is not specific to devicemapper. also, it seems hard to reproduce. — committed to AkihiroSuda/issues-docker by AkihiroSuda 8 years ago
- Remove docker/docker#5684 from the list because some different issues were discussed in the ticket and hence misleading — committed to AkihiroSuda/issues-docker by AkihiroSuda 8 years ago
This still is an issue for us (using 1.11.2 on Ubuntu 14.04.4 LTS (with KVM) (3.13.0-88-generic)).
Is there any open ticket I can subscribe to get updates?
@vieira you can unmount: umount /var/lib/docker/devicemapper/mnt/c39206003c7ae8992a554a9ac2ea130327fc4af1b2c389656c34baf9a56c84b5
and start the container again it should work
Still getting this too
I’ve worked around it by automatically unmounting any old mounts when the docker daemon comes back. I didn’t want to patch ubuntu’s big
/etc/init/docker.conf, so I put a small line in/etc/default/dockerinstead:cat /proc/mounts | grep "mapper/docker" | awk '{print $2}' | xargs -r umountThat seems to do the trick. I’ve combined it with having upstart manage starting and respawning of my actual docker containers, so now after a
service docker restart, all the containers will just come back.Linux zk1 3.10.0-327.28.3.el7.x86_64(centos 7) Docker version 1.12.1, build 23cf638
Error response from daemon: Driver devicemapper failed to remove root filesystem 228f2c2da3de4d5abd3881184aeb330a4c18e4311ecf404e2fb8cd4ffe15e901: devicemapper: Error running DeleteDevice dm_task_run failed
Bitten after after host going out-of-disk-space. Any command affecting the mount point hangs (including “docker ps”, “sync”, “ls <mntpnt>”, …)
Im seeing this in docker 1.12.3 on CentOs 7
dc2-elk-02:/root/staging/ls-helper$ docker --version Docker version 1.12.3, build 6b644ec dc2-elk-02:/root/staging/ls-helper$ uname -a Linux dc2-elk-02 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux dc2-elk-02:/root/staging/ls-helper$ docker rm ls-helper Error response from daemon: Driver devicemapper failed to remove root filesystem e1b9cdeb519d2f4bea53a552c8b76c1085650aa76c1fb90c8e22cac9c2e18830: Device is Busy
P.S. I am not using docker compose.
I am also having this problem. My setup: Ubuntu 14.04, updated to kernel “3.19.0-58-generic #64~14.04.1-Ubuntu”. Docker version 1.11.0, build 4dc5990. “docker-compose version 1.7.0, build 0d7bf73”.
When
docker-compose up -drestarts a container because of a new image, it often ends not being able to remove the stopped container.Only a reboot will help to be able to start the container. Then the problem is not 100% reproducable but it hapens very often. So I am forced to reboot the host machine frequently 😦
and
I had similar issue, I saw these error likes in my /var/log/syslog file:
Dec 16 14:32:18 rzing dockerd[3093]: time="2018-12-16T14:32:18.627417173+05:30" level=error msg="Failed to load container mount 00d7b9d64ff6c465276e67f5a5e3642ebacd9616c7602d4361b3a7fab038510a: mount does not exist" Dec 16 14:32:18 rzing dockerd[3093]: time="2018-12-16T14:32:18.627816711+05:30" level=error msg="Failed to load container mount fb108b942f8ed87a9e1affb6480ed477a8f5f823b2639e36348cde4a97924c5e: mount does not exist"I tried searching the mount point under/var/lib/docker/volumesbut didn’t find any finally reboot the system fixed the issueI also have the same issue in Docker version: 1.12.3
@chirangaalwis @kabobbob… I’m on RHEL 7.2 and kernel 3.10.
While stopping and starting containers using
docker-compose, I constantly get this error…The only workaround is to stop, and wait for a couple of seconds before trying again. The problem is that the restart is not guaranteed to to work. I sometimes try multiple times to restart.