moby: Docker does not free up disk space after container, volume and image removal

I don’t know if everyone is gone, but here are some tips and tricks. just make the docker system cleanup a chronjob: https://nickjanetakis.com/blog/docker-tip-32-automatically-clean-up-after-docker-daily

Start again by finding which is the culprit directory:

‘df -hx --max-depth=1 /’ and ‘df -h’

for me the culprit was docker: /var/lib/docker/overlay2/ short team this works: docker system prune -a -f

Long term: Run: ‘crontab -e’

insert this: ‘0 3 * * * /usr/bin/docker system prune -f’

@Kenji-K Not exactly. I stop the docker service nightly and rm -rf /var/lib/docker now, so at least it’s “stable”.

Imo this should be reopened.

“only having to nuke /var/lib/docker every few weeks” is not a satisfactory resolution for me.

Docker should stop leaking disk space if you regularly run system prune

@wannymiarelli that’s expected; images, containers, and volumes take up space. Have a look at docker system prune for easier cleanup than the command you showed.

I found a lot of unused images causing this issue, resolved running

docker rmi $(docker images -q)

We are running into this issue too:

# docker --version
Docker version 18.03.1-ce, build 9ee9f40

# docker system prune -a -f
Total reclaimed space: 0B
# docker volume prune -f
Total reclaimed space: 0B

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
# docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

# lsof | grep /var/lib/docker/overlay2

# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G   48K  1.8G   1% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
/dev/nvme0n1p1  493G  120G  372G  25% /
# du -sh /var/lib/docker
14G	/var/lib/docker
# du -h --max-depth 1 /var/lib/docker
4.0K	./trust
56K	./network
28K	./volumes
20K	./builder
14G	./overlay2
4.0K	./tmp
4.0K	./swarm
4.0K	./runtimes
120K	./containerd
20K	./plugins
201M	./image
4.0K	./containers
14G	.

I think this issue should be reopened. Let me know if there is any other info I can provide to debug this issue.

Thanks @neerolyte, let me go ahead and close this one 👍

@cpuguy83 @thaJeztah So just to clarify - is there actually some known safe way to clean up container and image data in any version of docker atm?

Because atm I’m stopping the service and just rm’ing stuff under the hood - but even with that I end up with overlay mounts dangling every now and then and have to actually reboot the box.

@thaJeztah it feels like the issue should be reopened 😄

I was able to recover 32 million inodes and 500GB of storage in /var/lib/docker/overlay2/ by removing unused GitLab Docker images and containers:

docker system prune -fa

I don’t want to reopen this issue, because became somewhat of a “kitchen sink” of “possibly related, but could be different issues”. Some issues were fixed, and other reports really need more details to investigate if there’s still issues to address. It’s better to start a fresh ticket (but feel free to link to this issue).

Looking at your earlier comment

But even after deleting that folder that shows up in ‘df’ (I didn’t care to recreate node-exporter, it was his folder) , the space didn’t clean up, and the mount for ‘/var’ shows this

First of all, I really don’t recommend manually removing directories from under /var/lib/docker as those are managed by the docker daemon, and removing files could easily mess up state. If that directory belonged to a running container, then removing it won’t actually remove the files until they’re no longer in use (and/or unmounted). See https://unix.stackexchange.com/a/68532

I see you’re mentioning you’re running node-exporter (https://github.com/prometheus/node_exporter), which can bind-mount the whole system’s filesystem into the container. (Possibly depending on mount-propagation settings), this can be problematic. If you bind-mount /var/lib/docker into a container, that bind-mount can “pull in” all mounts from other containers into that container’s mount namespace, which means that none of those container’s filesystems can be removed until the node-exporter container is removed (files unmounted). I believe there were also some kernel bugs that could result in those mounts to never be released.

As to differences between df and du, I’d have to do testing, but the combination of mounts being in different namespaces, together with overlay filesystems could easily lead to inconsistencies in reporting the actual space used (e.g. multiple containers sharing the same image would each mount the same image layers, but tools could traverse those mounts and account their size multiple times).

I see some mention of snaps and lxc in your comment; this could be unrelated, but if you installed docker using the snap packages, those packages are maintained by Canonical, and I’ve seen many reports of those being problematic; I’d recommend (if possible) to test if it also reproduces on the official packages (https://docs.docker.com/engine/install/ubuntu/)

(per earlier comments above); it’s possible that files are still in use (which could be by stopped containers or untagged images); in that case, the disk use may be legitimate; if possible, verify if disk space does go down after removing all images and containers.

If you think there’s a bug at hand, and have ways to reproduce the issue so that it can be looked into, feel free to open a new ticket, but when doing so;

• at least provide the details that are requested in the issue template
• provide exact steps to reproduce the issue (but try to write a minimal case to reproduce so that it could be used for writing a test if needed)

The only possible solution is to stop the service, then to delete the /var/lib/docker/* manually. Seriously this “product” never works correctly…

@thaJeztah

Also note, that removing containers with -f can result in layers being left behind; when using -f, docker currently ignores “filesystem in use” errors, and removes the container, even if it failed to remove the actual layers. This is a known issue currently, and being looked into to change that behavior

Could you please let me know of any actual bug numbers relating to this?

I should also mention that based on https://github.com/docker/docker/issues/24023 I’ve switched to running overlay2 (instead of the default of overlay 1 on CentOS 7). The issue exists against both overlay and overlay2, so I think it’s docker internals and not storage driver specific.

I’ve configured overlay2 by modifying daemon.json:

[root@*** ~]# cat /etc/docker/daemon.json
{
	"storage-driver": "overlay2",
	"storage-opts": [
		"overlay2.override_kernel_check=1"
	]
}

and yes I cleaned docker before starting up docker with the new driver by running rm -rf /var/lib/docker/*.

is it that all mounts defined by all containers will be locked by the one which mounts all /var/lib/docker or /var for example, and the disk space can be affected by that? (kernel or filesystem won’t be able to release disk space form files deleted in these folders or something?)

I’m a bit “hazy” on the exact details (I know @kolyshkin and @cpuguy83 dove more deeply into this when debugging some “nasty” situations), but my “layman explanation” is that “container A” has mounts in it’s own namespace (and thus only visible within that namespace), now if “container B” mounts those paths, those paths can only be unmounted if both “container B” and “container A” unmount them. But things can become more tricky than that; if “container A” has mounts with mount-propagation set (slave? shared?), the mounts of “container B” will also propagate to “container A”, and now there’s an “infinite loop” (container B’s mounts cannot be unmounted until container A’s mounts are unmounted, which cannot be unmounted until container A’s mounts are unmounted).

We have the same issue on docker Docker version 19.03.5, build 633a0ea838 the docker was restarted, container recreated (it had only node-exporter running!) , lsof can’t find any deleted open files, the container PID1 doesn’t show any open files. The ‘df’ was showing at and “du” thinks different about the usage, it doesn’t see the 82gb in use. But even after deleting that folder that shows up in ‘df’ (I didn’t care to recreate node-exporter, it was his folder) , the space didn’t clean up, and the mount for ‘/var’ shows this

mount | grep var
/dev/nvme1n1p1 on /var type ext4 (rw,noatime,data=ordered)
/var/lib/snapd/snaps/core_8268.snap on /snap/core/8268 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/amazon-ssm-agent_1480.snap on /snap/amazon-ssm-agent/1480 type squashfs (ro,nodev,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/var/lib/snapd/snaps/amazon-ssm-agent_1816.snap on /snap/amazon-ssm-agent/1816 type squashfs (ro,nodev,relatime)
/var/lib/snapd/snaps/core_8592.snap on /snap/core/8592 type squashfs (ro,nodev,relatime)

any ideas why “df” calculates used space in a way other tools can’t find the invisible files/ghosts? Should I inspect the filesystem somehow with advanced tools to understand what’s going on?

@ripcurld0 I don’t know. We don’t use CentOS or RHEL (ubuntu 16.04 latest patches), It happens with both AuFS and devicemapper (overlay don’t know) every time the partition goes out of space. We never use -f (and anyway if it is not safe for some reason, it should not be available). So yeah, nuking docker from time to time is currently our only option. But recently in another project with Docker Swarm we have to restart docker because the nodes were no longer able to communicate (even after rm / create the service). So I am guessing we’re getting used to downtime…

@ripcurld0 We’re running Ubuntu 16.04 w/ 4.9.0-040900-generic kernel, still seeing this issue.

I have the problem as well. It seems that when Docker encounters a “no space left on device error”, it is no longer able to reclaim space.

Just like @arturopie, I am running in the same issue:

# docker --version
Docker version 18.03.1-ce, build 9ee9f40

# docker system prune -a -f
Total reclaimed space: 0 B

docker volume prune -f
Total reclaimed space: 0 B

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
# docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

# lsof | grep /var/lib/docker/overlay2

# df -h
udev                        5.8G     0  5.8G   0% /dev
tmpfs                       1.2G  536K  1.2G   1% /run
/dev/mapper/aw--tests-root   90G   68G   18G  80% /
tmpfs                       5.9G     0  5.9G   0% /dev/shm
tmpfs                       5.0M     0  5.0M   0% /run/lock
tmpfs                       5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/xvda1                  228M  155M   61M  72% /boot
tmpfs                       1.2G     0  1.2G   0% /run/user/1002

sudo du -sh /var/lib/docker
1.4T	/var/lib/docker

sudo du -h --max-depth 1 /var/lib/docker
86M	/var/lib/docker/image
4.0K	/var/lib/docker/runtimes
1.4T	/var/lib/docker/overlay2
448K	/var/lib/docker/network
348K	/var/lib/docker/containerd
4.0K	/var/lib/docker/trust
20K	/var/lib/docker/plugins
333M	/var/lib/docker/volumes
211M	/var/lib/docker/swarm
4.0K	/var/lib/docker/tmp
4.0K	/var/lib/docker/containers
20K	/var/lib/docker/builder
1.4T	/var/lib/docker

I’m also not running k8s and I don’t have any loopback devices.

# losetup -a

What seems to be keeping this zombie disk space in use is that I have about 1500 mounts from elements in docker’s overlay filesystem. Here’s an exerpt:

# mount
overlay on /var/lib/docker/overlay2/4061c2b085c1bb9dc5c5eabe857446290f799ac9b7bf2645917c83cdb63b6d3b/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/4BMOONDLWUQ7ZWQXT3JORDKP7F:/var/lib/docker/overlay2/l/CGG6QC36WMFOWO6MAIHYWFYQ47:/var/lib/docker/overlay2/l/RYLPC32FMHRGH6VUZ4UDRVLSJV:/var/lib/docker/overlay2/l/VXZQOGGYK5C2KX27M7UAIZTCCQ:/var/lib/docker/overlay2/l/VXLVIIW6CAS2HNJO6NEHZLKBGM:/var/lib/docker/overlay2/l/QMMXIF2XL5PZJ5MFZXYBTK6RJS:/var/lib/docker/overlay2/l/4RNQ7THRJV7TWKOBAVC5XA5DEK,upperdir=/var/lib/docker/overlay2/4061c2b085c1bb9dc5c5eabe857446290f799ac9b7bf2645917c83cdb63b6d3b/diff,workdir=/var/lib/docker/overlay2/4061c2b085c1bb9dc5c5eabe857446290f799ac9b7bf2645917c83cdb63b6d3b/work)
overlay on /var/lib/docker/overlay2/a76b69897bd1b592fdb8790acd874e90863e8117b893703d8076f2b0a6b9fc13/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/4BMOONDLWUQ7ZWQXT3JORDKP7F:/var/lib/docker/overlay2/l/CGG6QC36WMFOWO6MAIHYWFYQ47:/var/lib/docker/overlay2/l/RYLPC32FMHRGH6VUZ4UDRVLSJV:/var/lib/docker/overlay2/l/VXZQOGGYK5C2KX27M7UAIZTCCQ:/var/lib/docker/overlay2/l/VXLVIIW6CAS2HNJO6NEHZLKBGM:/var/lib/docker/overlay2/l/QMMXIF2XL5PZJ5MFZXYBTK6RJS:/var/lib/docker/overlay2/l/4RNQ7THRJV7TWKOBAVC5XA5DEK,upperdir=/var/lib/docker/overlay2/a76b69897bd1b592fdb8790acd874e90863e8117b893703d8076f2b0a6b9fc13/diff,workdir=/var/lib/docker/overlay2/a76b69897bd1b592fdb8790acd874e90863e8117b893703d8076f2b0a6b9fc13/work)
overlay on /var/lib/docker/overlay2/9c3a878fe52f359b8e81540a8abb4ca5aaacf10c7d509be84d24e3dbc22a143b/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/4BMOONDLWUQ7ZWQXT3JORDKP7F:/var/lib/docker/overlay2/l/CGG6QC36WMFOWO6MAIHYWFYQ47:/var/lib/docker/overlay2/l/RYLPC32FMHRGH6VUZ4UDRVLSJV:/var/lib/docker/overlay2/l/VXZQOGGYK5C2KX27M7UAIZTCCQ:/var/lib/docker/overlay2/l/VXLVIIW6CAS2HNJO6NEHZLKBGM:/var/lib/docker/overlay2/l/QMMXIF2XL5PZJ5MFZXYBTK6RJS:/var/lib/docker/overlay2/l/4RNQ7THRJV7TWKOBAVC5XA5DEK,upperdir=/var/lib/docker/overlay2/9c3a878fe52f359b8e81540a8abb4ca5aaacf10c7d509be84d24e3dbc22a143b/diff,workdir=/var/lib/docker/overlay2/9c3a878fe52f359b8e81540a8abb4ca5aaacf10c7d509be84d24e3dbc22a143b/work)
overlay on /var/lib/docker/overlay2/96efd205e1fa795ab8f90c19aecab2cf71270a5901ff2546ca8926ac06925953/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/4BMOONDLWUQ7ZWQXT3JORDKP7F:/var/lib/docker/overlay2/l/CGG6QC36WMFOWO6MAIHYWFYQ47:/var/lib/docker/overlay2/l/RYLPC32FMHRGH6VUZ4UDRVLSJV:/var/lib/docker/overlay2/l/VXZQOGGYK5C2KX27M7UAIZTCCQ:/var/lib/docker/overlay2/l/VXLVIIW6CAS2HNJO6NEHZLKBGM:/var/lib/docker/overlay2/l/QMMXIF2XL5PZJ5MFZXYBTK6RJS:/var/lib/docker/overlay2/l/4RNQ7THRJV7TWKOBAVC5XA5DEK,upperdir=/var/lib/docker/overlay2/96efd205e1fa795ab8f90c19aecab2cf71270a5901ff2546ca8926ac06925953/diff,workdir=/var/lib/docker/overlay2/96efd205e1fa795ab8f90c19aecab2cf71270a5901ff2546ca8926ac06925953/work)

Unfortunately, even after cleaning up the mounts, no disk space is being freed and I still have to do a manual cleanup. The reboot doesn’t help.

# sudo umount -a --types overlay

@chr0n1x The only mounts that should exist in /var/lib/docker are mounts for running containers. If docker is not running there then you should be able to safely unmount them.

@frankh Sorry, I don’t think I was clear enough, I’m no longer running docker system prune. We only clear containers and untagged images regularly now - not everything else. When the stacks change under our build I think we’re just keeping some tags that no longer make sense - but the effort to “improve that” is higher than just leaving the existing (working) nuking process in place that kicks in when disk space gets low.

TLDR - if there is a bug still here, I no longer have a reliable process for reproducing it and even if I did, it’s not high on our list of issues.

rm -rf /var/lib/docker/tmp/* helped a bit (18GB of files of names like GetImageBlob537029535)

@jostyee FYI Ubuntu 16.04.3 LTS with 4.4.0-1030-aws kernel with aufs instead of overlay/overlay2 seems to run stable.

...
Server Version: 1.12.6
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 152
 Dirperm1 Supported: true
...

Don’t use docker rm -f… the -f causes Docker to ignore errors and remove the container from memory anyway. This should be better in 17.06 (fewer errors and -f no longer ignores errors).

Some quick questions;

• are you using docker-in-docker in your CI?
• if so, are you sharing the /var/lib/docker directory between the “docker in docker” container and the host?

Also note, that removing containers with -f can result in layers being left behind; when using -f, docker currently ignores “filesystem in use” errors, and removes the container, even if it failed to remove the actual layers. This is a known issue currently, and being looked into to change that behavior