moby: Unable to remove filesystem for xxx: remove /var/lib/docker/containers/xxx/shm: device or resource busy

Docker is unable to remove a container if another container has bind-mounted /var/lib/docker/containers. Only hits with older kernel, e.g. 3.13. Does not happen with newer kernel, e.g. 3.19. Also does not happen with Docker 1.8.

This is a repro carefully crafted to use only the simple alpine image, nothing fancy.

docker version:

Client:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   76d6bc9
 Built:        Tue Nov  3 19:20:09 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 1
Images: 93
Server Version: 1.9.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 95
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-55-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 490 MiB
Name: weave-gs-01
ID: LLWG:CZWC:AHZP:EA5W:5MG5:5IOC:C6GD:JKBJ:ZI2V:FMFT:ZNXG:G2OW
WARNING: No swap limit support

uname -a:

Linux weave-gs-01 3.13.0-55-generic #92-Ubuntu SMP Sun Jun 14 18:32:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.): VirtualBox VM in the above example; get identical symptoms from stock GCE RHEL 7.1.

How reproducible: 100% on older kernel; does not happen on newer kernel.

Steps to Reproduce:

$ docker run --name c1 -tdi alpine /bin/sh
3b1a8ae179c72c2b411e65a8e4099b61fbfd91f1843f86594bea51f56ca40766
$ docker run --name c2 -tdi -v /var/lib/docker/containers:/var/lib/docker/containers alpine /bin/sh
1938a9588bfd4708f82294ab9b2be509b6053a9802482565916ab403ae1fe729
$ docker rm -f c1

Actual Results:

Error response from daemon: Unable to remove filesystem for 3b1a8ae179c72c2b411e65a8e4099b61fbfd91f1843f86594bea51f56ca40766: remove /var/lib/docker/containers/3b1a8ae179c72c2b411e65a8e4099b61fbfd91f1843f86594bea51f56ca40766/shm: device or resource busy
Error: failed to remove containers: [c1]

Expected Results:

c1

Additional info: This is similar to #17823, although it only hits on older kernels.

About this issue

Original URL
State: closed
Created 9 years ago
Reactions: 15
Comments: 23 (5 by maintainers)

Commits related to this issue

Work around https://github.com/docker/docker/issues/17902 — committed to mantl/mantl by langston-barrett 8 years ago
Run cAdvisor directly on host See Docker issue #17902 [1] for more information. [1] https://github.com/docker/docker/issues/17902 — committed to martin-helmich/salt-microservices by martin-helmich 8 years ago

Most upvoted comments

Just wanted to say thank you for identifying this issue. It has been driving me nuts for the last couple of days!

jeinwag on Mar 7, 2016

@sakserv No promises but this script tries to count references of the shm mount per container. 1 is normal. Any more than this and you may be nesting mounts. For instance here e5365f791a8d has no mounts and e9fd794979a0: mounts /var/lib/docker.

$ bash shmcounter.sh 
e9fd794979a0: 3
e5365f791a8d: 1

False positives are plausible given that this is a twenty minute hack but if it works for you maybe it will give you a pointer.

#!/bin/bash

containers=($(docker ps --format="{{ .ID }}"))

for id in "${containers[@]}"; do
  pid="$(docker top "$id" | tail -n+2 | head -n1 | awk '{ print $2 }')"
  count="$(findmnt --task "$pid" | grep -c /shm)"
  echo "$id: $count"
done

mikesimons on Feb 2, 2017

This is fixed in 17.12.1 and up. Thanks!

cpuguy83 on May 3, 2018