moby: Container removal broken with ZFS as of 17.11.0-ce

Description

As of Docker 17.11.0-ce container removal is broken when used with the ZFS storage driver.

Steps to reproduce the issue:

  1. docker run -it --name bug1 --rm debian bash in one terminal window
  2. docker run -it --name bug2 --rm debian bash in another terminal window
  3. docker stop bug1 in a third terminal window
  4. docker ps and notice Status = Removal In Progress
  5. docker rm bug1
  6. grep in /proc/*/mounts
  7. Notice that docker-containerd-shim (of the other container) holds the mount.

Describe the results you received:

[timwolla@~]docker rm bug1
Error response from daemon: driver "zfs" failed to remove root filesystem for 0ba97ff3ff17317fce4bbe3ae6cc63fab0247a9689b4ee26f1ff6d2c0d4408bc: exit status 1: "/sbin/zfs zfs destroy -r scratch/docker/7a0e73a9714f4a9ebf8bda56ca01c026a7ce29c1275900403ce99b55038e1834" => cannot destroy 'scratch/docker/7a0e73a9714f4a9ebf8bda56ca01c026a7ce29c1275900403ce99b55038e1834': dataset is busy
[timwolla@~ [1]]grep 7a0e73a9714f4a9ebf8bda56ca01c026a7ce29c1275900403ce99b55038e1834 /proc/*/mounts
/proc/8625/mounts:scratch/docker/7a0e73a9714f4a9ebf8bda56ca01c026a7ce29c1275900403ce99b55038e1834 /scratch/docker/zfs/graph/7a0e73a9714f4a9ebf8bda56ca01c026a7ce29c1275900403ce99b55038e1834 zfs rw,relatime,xattr,noacl 0 0
[timwolla@~]ps aux |grep 8625
root      8625  0.0  0.1 438744 13208 ?        Sl   19:26   0:00 docker-containerd-shim --namespace moby --workdir /scratch/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/e62dc25804ced0d0a222d3eca67210201bd8ddf3c17756a03b9bbedc796a2cc7 --address /var/run/docker/containerd/docker-containerd.sock --runtime-root /var/run/docker/runtime-runc
timwolla  8871  0.0  0.0  15776   984 pts/22   S+   19:27   0:00 grep --color=auto 8625

Describe the results you expected:

I expected a successful removal of the container.

Additional information you deem important (e.g. issue happens only occasionally):

Can be reliably reproduced as of Docker 17.11. My unit file is modified as follows:

[Unit]
Requires=zfs.target
After=zfs.target

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --icc=true -g /scratch/docker/ -H fd:// --storage-driver=zfs --live-restore --init

Output of docker version:

Client:
 Version:      17.11.0-ce
 API version:  1.34
 Go version:   go1.8.3
 Git commit:   1caf76c
 Built:        Mon Nov 20 18:37:39 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.11.0-ce
 API version:  1.34 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   1caf76c
 Built:        Mon Nov 20 18:36:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 173
Server Version: 17.11.0-ce
Storage Driver: zfs
 Zpool: scratch
 Zpool Health: ONLINE
 Parent Dataset: scratch/docker
 Space Used By Parent: 9999368192
 Space Available: 16792281088
 Parent Quota: no
 Compression: on
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 992280e8e265f491f7a624ab82f3e238be086e49
runc version: 0351df1c5a66838d0c392b4ac4cf9450de844e2d
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-101-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.773GiB
Name: *snip*
ID: *snip*
Docker Root Dir: /scratch/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

My developer Laptop, running Ubuntu 16.04 LTS.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 5
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

https://github.com/moby/moby/pull/35674 was merged; I’ll go ahead and close this one