moby: 'docker system df' does not list all space being used

For me docker system df -v shows:

Images space usage:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
nuccy               latest              eb40b1ff80bc        About an hour ago   20.37GB             53.7MB              20.32GB             0
debian              stretch             d508d16c64cd        4 weeks ago         100.6MB             0B                  100.6MB             0
bitnami/minideb     stretch             fa89f2915564        5 weeks ago         53.7MB              53.7MB              0B                  0

Containers space usage:

CONTAINER ID        IMAGE               COMMAND             LOCAL VOLUMES       SIZE                CREATED             STATUS              NAMES

Local Volumes space usage:

VOLUME NAME                                                        LINKS               SIZE
6aec75524e3dedd97dca14c77a7d0b2ed6c52bf08b8626dfcfb1e7d6704a3bb3   0                   0B
a9fe3492b569a6fd5d27af16f614d6a82a2ab332d948f924e644350313d37706   0                   0B
sv-cfg                                                             0                   597.3kB

Build cache usage: 35.05MB

CACHE ID            CACHE TYPE          SIZE                CREATED             LAST USED           USAGE               SHARED
857e4ad1a7b8        regular             1.62MB              6 weeks ago                             0                   false
13d5529fd232        regular             101MB               2 weeks ago         2 weeks ago         12                  true
6659c036b8dc        internal            880kB               6 weeks ago         2 weeks ago         27                  false
8c934103363f        frontend            16.3MB              6 weeks ago         13 days ago         44                  false
ca8bf53712bd        regular             53.7MB              2 weeks ago         8 hours ago         8                   true
00deee01db1f        frontend            16.3MB              10 days ago         8 hours ago         4                   false

So about 20 GB of space being used. However the directory /var/lib/docker/overlay2 indicates there are more images, meaning that about 40 GB of space are being used by docker:

root /var/lib/docker/overlay2 # du -hsx * | sort -rh
19G     fd8a111f864daa966c7cdde633e81ca9f0c6a6b085a01b699442f1dd2ae23707
17G     81b29726d152a815fa118513ea844e9aee654111185bb020461f8fdc5ff363cd
245M    8975705b1d4c7d7d7132d9d8f1c2b20cd393e18b262e16d6108335ba1c7b2f2b
111M    c3ab47614a0b32adbf20d6a246e0e07d6b7d3260d18b24320acccab076ac3f53
61M     44c04b2edb92b6b7f4ffed950da58b3a990e9fd2c0fa80f098340f4309abfdf3
50M     74168178318e1abff6f59d96aed3995e7e67ae211ea6b96c37ff5c2d7226e695
16M     f30cb0f43677d828528edf31d8c7b41041f5ed23d4ec15258446a9fa615d61dc
16M     37ffdb7679240e4c31634aa5ec9c7a7b6a8c8f4fecb5acde7628cdbc4d4ac7c5
6,1M    2f218061dc4f061e6dd625c47d96ba2f2cb5e5aa791ec65dd59728cc8ca7186c
2,4M    eb31594cba19cc0dd48a684c869be28832ce940d1dd3612445e4ade2c711a1d9
1,7M    c8bdbe9b03b4fb9bda5d86df450a911a87a5b51fe8149bfcc5e40f32860f243c
884K    04ead42412872adf89de4ee1293f45d15c6bc3b66f786ff78e26f0de737bc0df
76K     l
44K     39dd1d95705e295e0a5cded06e7c858fc6010fcc5c52599717ba7108b6071f1f
40K     d6ff20865169fc9c11d854760e9f7e0476a522cec83801faa6f2bc8a19697d18
36K     75006f6885eb22a524cc1fdb8543e30e00464e211a6d3aba25610d46ea98d421
28K     64c65b9b4796cd0e9d1c23c846c74c3a1a017b0f074571e75ab470e4b4a3f8a1

A docker system prune wouldn’t prune those images either.

I had noticed this when my server ran out of space and docker system df still showed a moderate usage - which is simply not true.

I’m on CentOS 7.

# docker version
Client:
 Version:           18.09.3
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        774a1f4
 Built:             Thu Feb 28 06:33:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.3
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       774a1f4
  Built:            Thu Feb 28 06:02:24 2019
  OS/Arch:          linux/amd64
  Experimental:     true

#30975 related?

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 8
Comments: 35 (9 by maintainers)

Most upvoted comments

Well the issue seems to be that devs see the issue in userspace reporting incorrect space left on the device.

Those arguments dont hold up however, if my HDD is 100% full it’s 100% full. I don’t understand why my bug report (and the ones of many others) do not get taken seriously.

There is obviously something wrong and it needs to be taken seriously and adressed properly.

+18

Nuc1eoN on Sep 6, 2020

I’m having a similar problem. After deleting everything (docker rm + docker rmi) and doing a system prune, this is what I’m getting:

# docker version
Client: Docker Engine - Community
 Version:           19.03.13
[snip]
Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
[snip]
# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              0                   0                   0B                  0B
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

# df -h /var/lib/docker/
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-lv_docker  196G   88G  109G  45% /var/lib/docker

I notice that buildkit is leaving a 23 Gb metadata.db file, despite reporting used space as 0. And I would expect this metadata.db file to have been cleaned out by “docker system prune -a”. Or by a “docker service restart”. Tried both, neither worked. There is also a bunch of folders under /var/lib/docker/buildkit/net, reported to be 16Kb each for a total of about 380Mb.

# ls -lah /var/lib/docker/buildkit
total 23G
drwx--x--x     5 root root 4.0K Jun 15 13:18 .
drwx--x--x    15 root root 4.0K Nov 19 13:25 ..
-rw-------     1 root root  46M Nov 19 13:25 cache.db
drwxr-xr-x     4 root root 4.0K Jun 15 13:18 content
drwx--x--x     2 root root 4.0K Nov 19 13:25 executor
-rw-------     1 root root  23G Nov 19 13:21 metadata.db
drwxr-xr-x 22217 root root 1.2M Nov 19 12:18 net
-rw-------     1 root root 2.0M Nov 19 13:21 snapshots.db

The rest of the space is mostly used under overlay2/:

# du -h --max-depth=1 /var/lib/docker
24G     ./buildkit
4.0K    ./containers
20K     ./plugins
4.0K    ./trust
4.0K    ./tmp
16K     ./lost+found
65G     ./overlay2
480M    ./image
48K     ./volumes
4.0K    ./swarm
20K     ./builder
4.0K    ./runtimes
92K     ./network
88G     .

To solve the situation I had to stop the docker service, unmount the /var/lib/docker filesystem, mkfs, remount, and restart the service.

lbndev on Nov 19, 2020

Not 100% sure, but I think the -x / --one-file-system option does that, and ignores mounts;

With --one-file-system;

du --max-depth=1 --one-file-system -ch /var/lib/docker/
72K	/var/lib/docker/buildkit
52K	/var/lib/docker/network
4.0K	/var/lib/docker/runtimes
804K	/var/lib/docker/image
28K	/var/lib/docker/volumes
4.0K	/var/lib/docker/trust
80M	/var/lib/docker/overlay2
4.0K	/var/lib/docker/tmp
156K	/var/lib/docker/containers
20K	/var/lib/docker/plugins
4.0K	/var/lib/docker/swarm
20K	/var/lib/docker/builder
81M	/var/lib/docker/
81M	total

Without;

du --max-depth=1 -ch /var/lib/docker/
72K	/var/lib/docker/buildkit
52K	/var/lib/docker/network
4.0K	/var/lib/docker/runtimes
804K	/var/lib/docker/image
28K	/var/lib/docker/volumes
4.0K	/var/lib/docker/trust
237M	/var/lib/docker/overlay2
4.0K	/var/lib/docker/tmp
156K	/var/lib/docker/containers
20K	/var/lib/docker/plugins
4.0K	/var/lib/docker/swarm
20K	/var/lib/docker/builder
239M	/var/lib/docker/
239M	total

thaJeztah on Aug 11, 2020

At least in our case container log files are clearly not the issue. All commands executed as root

docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          273       4         104.3GB   93.91GB (90%)
Containers      26        25        479.6kB   0B (0%)
Local Volumes   95        95        20.76MB   0B (0%)
Build Cache     2071      0         2.706MB   2.706MB

du --max-depth=1 -ch /var/lib/docker/
4,0K    /var/lib/docker/swarm
29M     /var/lib/docker/buildkit
4,0K    /var/lib/docker/runtimes
23M     /var/lib/docker/containers
23M     /var/lib/docker/volumes
4,0K    /var/lib/docker/trust
807M    /var/lib/docker/image
6,3M    /var/lib/docker/tmp
297G    /var/lib/docker/overlay2
16K     /var/lib/docker/plugins
180K    /var/lib/docker/network
298G    /var/lib/docker/
298G    total

du --max-depth=1 --one-file-system -ch /var/lib/docker/
4,0K    /var/lib/docker/swarm
29M     /var/lib/docker/buildkit
4,0K    /var/lib/docker/runtimes
23M     /var/lib/docker/containers
23M     /var/lib/docker/volumes
4,0K    /var/lib/docker/trust
807M    /var/lib/docker/image
6,3M    /var/lib/docker/tmp
221G    /var/lib/docker/overlay2
16K     /var/lib/docker/plugins
180K    /var/lib/docker/network
222G    /var/lib/docker/
222G    total

ncdu /var/lib/docker
--- /var/lib/docker ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
. 260,2 GiB [##########] /overlay2                                                                                                                                                          
  806,6 MiB [          ] /image
   28,1 MiB [          ] /buildkit
   21,7 MiB [          ] /containers
   21,1 MiB [          ] /volumes
    6,2 MiB [          ] /tmp
e   4,0 KiB [          ] /trust
e   4,0 KiB [          ] /swarm
e   4,0 KiB [          ] /runtimes
!   0,0   B [          ]  plugins
!   0,0   B [          ]  network

ncdu -x /var/lib/docker
--- /var/lib/docker ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
. 220,7 GiB [##########] /overlay2                                                                                                                                                          
  806,6 MiB [          ] /image
   28,1 MiB [          ] /buildkit
   22,0 MiB [          ] /volumes
   20,4 MiB [          ] /containers
    6,2 MiB [          ] /tmp
  180,0 KiB [          ] /network
   16,0 KiB [          ] /plugins
e   4,0 KiB [          ] /trust
e   4,0 KiB [          ] /swarm
e   4,0 KiB [          ] /runtimes

albertca on May 4, 2022

Per the requirements for opening a new issue when there is one already outlining my issue. I am experiencing the same issue as @Nuc1eoN

docker system df shows a very different totals list from what is the actual size of /var/lib/docker/overlay2.

My details are as follows:

OS:

NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Docker Info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 3649
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1083-aws
Operating System: Ubuntu 16.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.08GiB
Name: i-0991227e95444b0c7
ID: T6FQ:TO3I:B4LS:S2XX:KQMQ:L77H:ZY25:IEFY:RJ62:FCHN:2G4T:XI5P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

All Existing Containers:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                   PORTS               NAMES
68e770f9e0e7        938f2e0db3d0        "/bin/sh -c bin/prec…"   3 hours ago         Exited (1) 3 hours ago                       zen_goldberg

Docker System DF Output:

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              1218                1                   444.9GB             444.1GB (99%)
Containers          1                   0                   83.73MB             83.73MB (100%)
Local Volumes       8                   0                   149.1MB             149.1MB (100%)
Build Cache         181                 0                   22.23MB             22.23MB

NOTE: We use BuildKit, so we have build cache objects.

Host DF Output:

Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  337M  2.8G  11% /run
/dev/nvme0n1p1  970G  762G  208G  79% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1335
/dev/loop3       89M   89M     0 100% /snap/core/7169
/dev/loop0       89M   89M     0 100% /snap/core/7270
/dev/loop2       18M   18M     0 100% /snap/amazon-ssm-agent/1455
tmpfs           3.2G     0  3.2G   0% /run/user/10000
tmpfs           3.2G     0  3.2G   0% /run/user/1006

NOTE: /var/lib/docker/ is under our root partition on /, which is /dev/nvme0n1p1, which brings another question… why is overlay not showing up as a mount in the df -h output?

And here is the final bit, showing the output of ncdu -x on /var/lib/docker/overlay2 (I am not listing every single dir line in the output… it would crush this Github page… but instead am posting the header dir and the final total report):

Screenshot: The total size reported by ncdu -x for the overlay2 directory is 678.1GB, whereas docker system df shows a combined total of around 445GB.

phantasm66 on Jul 22, 2019

Also make shure to exclude container log files from that, as they’re not included in docker’s size output

thaJeztah on May 4, 2022

What else, apart from log data, is relevant and not taken into account? Could it be that Moby/Docker somehow got lost on what belongs to it and therefore not cleaning it in /var/lib/docker & not counting it on docker system df? Or could it be related to issue 21925 (Docker does not free up disk space after container, volume and image removal)?

Currently we have a huge difference. /var/lib/docker/ is more than twice the size of what docker system df reports (33G):

# du --max-depth=1 -ch /var/lib/docker/
0	/var/lib/docker/tmp
0	/var/lib/docker/containers
0	/var/lib/docker/runtimes
0	/var/lib/docker/plugins
5.7G	/var/lib/docker/overlay2
24M	/var/lib/docker/image
24K	/var/lib/docker/volumes
0	/var/lib/docker/trust
764K	/var/lib/docker/network
62M	/var/lib/docker/swarm
16K	/var/lib/docker/builder
56K	/var/lib/docker/buildkit
27G	/var/lib/docker/4099.500000
33G	/var/lib/docker/
33G	total

but docker shows a bit more than 12.58GB:

# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              13                  13                  12.58GB             1.129GB (8%)
Containers          24                  24                  5.093MB             0B (0%)
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

Meaning, that /var/lib/docker/ contains more than double the size docker is reporting.

And there aren't any huge logs either (click to expand)

du -a -BM /var/lib/docker/ | sort -n -r | grep -i '\-json.log'
31M	/var/lib/docker/4099.500000/containers/e49df7f0f75e13e54f7dcf4b93841cea0186dac4c0b9f5754992850cd69cb9b6/e49df7f0f75e13e54f7dcf4b93841cea0186dac4c0b9f5754992850cd69cb9b6-json.log
1M	/var/lib/docker/4099.500000/containers/f95cb736dee7e8b4d38cf7d745e85c63ada15c0348c5883969e06896414bdaa2/f95cb736dee7e8b4d38cf7d745e85c63ada15c0348c5883969e06896414bdaa2-json.log
1M	/var/lib/docker/4099.500000/containers/f10fc12d30ba7c483f2e740f83baded05671ae97c4295dc1a5f9fcf01909f8f9/f10fc12d30ba7c483f2e740f83baded05671ae97c4295dc1a5f9fcf01909f8f9-json.log
1M	/var/lib/docker/4099.500000/containers/e1ab91fe9f96ba16af957ed34027655e7df81173a72b7090bb08d47fb45fab97/e1ab91fe9f96ba16af957ed34027655e7df81173a72b7090bb08d47fb45fab97-json.log
1M	/var/lib/docker/4099.500000/containers/e0700c6ec160b4c6fd0f70dd0f980f00ba3237fa26ad28e9e84e2fe872687528/e0700c6ec160b4c6fd0f70dd0f980f00ba3237fa26ad28e9e84e2fe872687528-json.log
1M	/var/lib/docker/4099.500000/containers/c46527c60e866e8879e10bdf876bd7be3089777af094b11e671332d1bff78975/c46527c60e866e8879e10bdf876bd7be3089777af094b11e671332d1bff78975-json.log
1M	/var/lib/docker/4099.500000/containers/c414c1c178d2bed6e3eb606e9b6d5380df58ebbf93dd8729e2f6b86f8143241c/c414c1c178d2bed6e3eb606e9b6d5380df58ebbf93dd8729e2f6b86f8143241c-json.log
1M	/var/lib/docker/4099.500000/containers/bdf71e39440d159fd5181cbfa13cfc205436b3d744657a26cce71546d8334724/bdf71e39440d159fd5181cbfa13cfc205436b3d744657a26cce71546d8334724-json.log
1M	/var/lib/docker/4099.500000/containers/a8e918b5a3d82891d8f6608586223e3522f26335a11c790f6a0829db6d416acc/a8e918b5a3d82891d8f6608586223e3522f26335a11c790f6a0829db6d416acc-json.log
1M	/var/lib/docker/4099.500000/containers/9652b82c111997359670a597761b0d1a2485b0d20712a8d8ac2a9d92f8ffe05f/9652b82c111997359670a597761b0d1a2485b0d20712a8d8ac2a9d92f8ffe05f-json.log
1M	/var/lib/docker/4099.500000/containers/911ad637ef3002da681b88e6537daf51f4e1b3872e23765bdfbf84f18b2a96e9/911ad637ef3002da681b88e6537daf51f4e1b3872e23765bdfbf84f18b2a96e9-json.log
1M	/var/lib/docker/4099.500000/containers/8f41a5fba0d22713da1d8290febfbd3e12dd4d1d3cd5ae20160abc8d3940694f/8f41a5fba0d22713da1d8290febfbd3e12dd4d1d3cd5ae20160abc8d3940694f-json.log
1M	/var/lib/docker/4099.500000/containers/77ac2a5fb6fc9731eeef2d8c03f2c2aec2c88449979325ba0e552eec37f54a6d/77ac2a5fb6fc9731eeef2d8c03f2c2aec2c88449979325ba0e552eec37f54a6d-json.log
1M	/var/lib/docker/4099.500000/containers/62f95ae71278b1c397d5fc3e4a1908beaafe824ea2ed6f0e73cf31419b93e95f/62f95ae71278b1c397d5fc3e4a1908beaafe824ea2ed6f0e73cf31419b93e95f-json.log
1M	/var/lib/docker/4099.500000/containers/4a73ed2ab9361264f27c02bf8889a9f5ce617d93ff15e3e237202bc6ceb84452/4a73ed2ab9361264f27c02bf8889a9f5ce617d93ff15e3e237202bc6ceb84452-json.log
1M	/var/lib/docker/4099.500000/containers/323d41d8ce37fde57202553f96c46fe908b73129a4ee77fbdfbacbae140bd2bd/323d41d8ce37fde57202553f96c46fe908b73129a4ee77fbdfbacbae140bd2bd-json.log
1M	/var/lib/docker/4099.500000/containers/2b819c3e43ba39f3df4eb23d74946aaa90154cdd02c316b500a5006462f3dc4f/2b819c3e43ba39f3df4eb23d74946aaa90154cdd02c316b500a5006462f3dc4f-json.log
1M	/var/lib/docker/4099.500000/containers/1eefd0354cd8b90023080fd0055f3d846aee04627ba125db9944781dd9ef955f/1eefd0354cd8b90023080fd0055f3d846aee04627ba125db9944781dd9ef955f-json.log
1M	/var/lib/docker/4099.500000/containers/05e6ad3f5120b33bb1fa402a498100683b2306a953a7f66c82090c92e02cf475/05e6ad3f5120b33bb1fa402a498100683b2306a953a7f66c82090c92e02cf475-json.log
0M	/var/lib/docker/4099.500000/containers/cdc0681d8dda5340a3ed66878dc23c27d09b1a5f19de84045e23f470a680accd/cdc0681d8dda5340a3ed66878dc23c27d09b1a5f19de84045e23f470a680accd-json.log
0M	/var/lib/docker/4099.500000/containers/bec2451a8372d7bd7203c93605954b2649d5a21dbcd35d06f6c713274b2d4e7b/bec2451a8372d7bd7203c93605954b2649d5a21dbcd35d06f6c713274b2d4e7b-json.log
0M	/var/lib/docker/4099.500000/containers/6a9cc45e4cc6fbc93e9940874f229de6277c968dab5de1d5c449f4412997cf31/6a9cc45e4cc6fbc93e9940874f229de6277c968dab5de1d5c449f4412997cf31-json.log
0M	/var/lib/docker/4099.500000/containers/3f36510bbe686ef663d74b65e0aa7527cb9ad167a25c24b168e188a5a6690117/3f36510bbe686ef663d74b65e0aa7527cb9ad167a25c24b168e188a5a6690117-json.log
0M	/var/lib/docker/4099.500000/containers/0f525f6cd30000ee90d4f58c796b69ba542c43b4efff5c11540e26e94734415d/0f525f6cd30000ee90d4f58c796b69ba542c43b4efff5c11540e26e94734415d-json.log

Docker version (click to expand)

Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        42e35e61f3
 Built:             Mon Jun  1 09:12:22 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       42e35e61f3
  Built:            Mon Jun  1 09:10:54 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Docker is running within a user namespace (hence /var/lib/docker/4099.500000/) and this is on Ubuntu 18.04.4 LTS: 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

du --max-depth=1 -ch /var/lib/docker/4099.500000 (click to expand)

33M	/var/lib/docker/4099.500000/containers
0	/var/lib/docker/4099.500000/plugins
27G	/var/lib/docker/4099.500000/overlay2
41M	/var/lib/docker/4099.500000/image
24K	/var/lib/docker/4099.500000/volumes
0	/var/lib/docker/4099.500000/trust
252K	/var/lib/docker/4099.500000/network
124M	/var/lib/docker/4099.500000/swarm
16K	/var/lib/docker/4099.500000/builder
56K	/var/lib/docker/4099.500000/buildkit
0	/var/lib/docker/4099.500000/tmp
0	/var/lib/docker/4099.500000/runtimes
27G	/var/lib/docker/4099.500000
27G	total

Apart from the issue with inaccurate results: which was the first release containing the garbage collector? Could the garbage collector solve the issue with the growing space?

There a lot of entries that are neither containers nor images any more. I checked it this way (got the idea from these comments):

$ docker inspect $(docker ps -qa) | grep --only-matching --extended-regexp '[a-f0-9]{64}' >> inspect-hashs.txt
$ docker inspect $(docker images -qa) | grep --only-matching --extended-regexp '[a-f0-9]{64}' >> inspect-hashs.txt
$ ls -l /var/lib/docker/overlay2/ > overlays.txt
$ grep --invert-match --line-regexp --fixed-strings -f inspect-hashs.txt overlays.txt | grep -oE '[a-f0-9]{64}'

The result gave 77 entries. Could this be related? And the size seems to be significant, too. I checked it with

$ grep --invert-match --line-regexp --fixed-strings -f inspect-hashs.txt overlays.txt | grep -oE '[a-f0-9]{64}' >  orphaned
$ cat orphaned | while read dir; do du -c -d0 "/var/lib/docker/overlay2/$dir/" | tail -n1; done | awk '{x+=$1} END {print x}'

PS: This seems to be a duplicate of Issue #942. So this issue might be a bit older. PPS: Could the issue of the growing storage be related to #32360? PPPS: These problems also seems to be described on StackOverflow. For instance (here or here).

alexanderadam on Aug 6, 2020

du will not understand that container filesystems are overlay mounts and not actually taking up extra space.

cpuguy83 on Mar 10, 2019