moby: Delete data in a container devicemapper can not free used space

Environment

[root@localhost Desktop]# cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core) 

[root@localhost Desktop]# uname -a
Linux localhost.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015         x86_64 x86_64 x86_64 GNU/Linux

[root@localhost Desktop]# docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

[root@localhost Desktop]# docker info
Containers: 0
Images: 0
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:1-24931-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 53.74 MB
 Data Space Total: 107.4 GB
 Data Space Available: 35.46 GB
 Metadata Space Used: 606.2 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 1.784 GiB
Name: localhost.localdomain
ID: K3C5:7YDM:RIXL:3SFO:FNGY:5N5E:44BM:Z72Y:O6N6:RYU5:ZOWQ:XWR3
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Reproduced

[root@localhost Desktop]# docker pull busybox

[root@localhost Desktop]# docker info
Containers: 0
Images: 2
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:1-24931-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 61.01 MB
 Data Space Total: 107.4 GB
 Data Space Available: 35.45 GB
 Metadata Space Used: 626.7 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 1.784 GiB
Name: localhost.localdomain
ID: K3C5:7YDM:RIXL:3SFO:FNGY:5N5E:44BM:Z72Y:O6N6:RYU5:ZOWQ:XWR3
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Write a file in container

[root@localhost Desktop]# docker run -it busybox
/ # dd bs=1M count=1024 if=/dev/zero of=test
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 1.378665 seconds, 742.7MB/s

[root@localhost Desktop]# docker info
Containers: 1
Images: 2
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:1-24931-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.14 GB
 Data Space Total: 107.4 GB
 Data Space Available: 34.37 GB
 Metadata Space Used: 1.176 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 1.784 GiB
Name: localhost.localdomain
ID: K3C5:7YDM:RIXL:3SFO:FNGY:5N5E:44BM:Z72Y:O6N6:RYU5:ZOWQ:XWR3
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

After delete it, the used space can not free

[root@localhost Desktop]# docker run -it busybox
/ # dd bs=1M count=1024 if=/dev/zero of=test
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 1.378665 seconds, 742.7MB/s
/ # rm -f test 

[root@localhost Desktop]# docker info
Containers: 1
Images: 2
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:1-24931-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.14 GB
 Data Space Total: 107.4 GB
 Data Space Available: 34.37 GB
 Metadata Space Used: 1.176 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 1.784 GiB
Name: localhost.localdomain
ID: K3C5:7YDM:RIXL:3SFO:FNGY:5N5E:44BM:Z72Y:O6N6:RYU5:ZOWQ:XWR3
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Other information

Delete images or container, the space wil be reclaimed

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 28 (17 by maintainers)

Most upvoted comments

running fstrim /proc/$(docker inspect --format='{{ .State.Pid }}' <cid>)/root on host works for me.

centos 7.2 + docker 1.9.1(devicemapper, xfs)

+45
depay on Jun 2, 2016

Awesome! @depay’s solution worked perfectly! Here’s a version to apply this for all containers:

# docker ps -qa | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ fstrim /proc/Z/root/

Before:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G   28G  378M  99% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  394M  3.6G  10% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup

After:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G   16G   13G  55% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  394M  3.6G  10% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
+24
Umkus on Jul 13, 2016

To me, it is best to move away from loop devices. Not only these are slow but these will introduce the issues of discards.

If you just can’t move away from loop devices, then one has to either issue fstrim at regular intervals or enable online discard in container filesystem. (-o discard).

Right now in the code, for ext4 nodiscard is hard coded. So I don’t think there is a way to enable online discards for ext4. But one should be able to do so for xfs. Try following.

  • Remove existing instance of docker metadata rm -rf /var/lib/docker
  • Pass following options to docker daemon. -s devicemapper --storage-opt dm.fs=xfs --storage-opt dm.mountopt=discard
  • Start docker daemon.

Now if you remove the file, fs should issue discards and reclaim the space back. This should be slower though as discards are issues inline.

+2
rhvgoyal on Jan 5, 2016

@aekazitt that’s because you have stopped containers, running docker container prune beforehand should fix it (or you could put a grep -Ev '^0$' before xargs).

+1
L-P on Sep 24, 2018

When I run the command above, it does not solve the issue. This command:

docker ps -qa | xargs docker inspect --format='{{ .State.Pid }}' | xargs -IZ fstrim /proc/Z/root/

Instead I get this response

fstrim: cannot open /proc/0/root: No such file or directory

The distribution version is below, running on AWS Beanstalk:

Linux version 4.14.62-65.117.amzn1.x86_64 (mockbuild@gobi-build-60009) (gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC))

Docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.4
 Git commit:   3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
 Built:        Fri Jul 20 19:24:06 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   7390fc6/18.03.1-ce
  Built:        Fri Jul 20 19:25:23 2018
  OS/Arch:      linux/amd64
  Experimental: false
+1
aekasitt on Aug 30, 2018
Read more comments on GitHub
← moby: Deinstall and install of docker ce fails on ubuntu 18.04
moby: Device-mapper does not release free space from removed images →