moby: docker storage drivers devicemapper and btrfs very slow

Description of problem:

Docker with devicemapper/btrfs is order of magnitude slower than with AUFS.

docker version:

Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): d344625
OS/Arch (client): linux/amd64
Server version: 1.3.3
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): d344625

docker -D info:

Containers: 0
Images: 19
Storage Driver: devicemapper
 Pool Name: docker-8:18-402882181-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 465.4 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 1.204 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
Debug mode (server): false
Debug mode (client): true
Fds: 10
Goroutines: 11
EventsListeners: 0
Init SHA1: 623458afefaf342877bebd84ba389315167962ca
Init Path: /usr/lib/docker.io/dockerinit
WARNING: No memory limit support
WARNING: No swap limit support

uname -a:

Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux

Environment details:

Physical, Debian GNU/Linux 8.0 (jessie). SSD: OCZ-VERTEX2, HDD: WDC WD7501AALS-00J7B0.

How reproducible:

Always

Steps to Reproduce:

$ cat >Dockerfile <<EOF
FROM debian:jessie
ENV A 1
ENV B 2
ENV C 3
ENV D 4
ENV E 5
ENV F 6
ENV G 7
ENV H 8
EOF

1. sudo docker info | grep Storage
2. run once to download base image: sudo docker build -t test --no-cache .
3. run again to measure just the ENVs: sudo time docker build -t test --no-cache .

Actual Results: Noticeable delay

Storage Driver: devicemapper
[...]
0.02user 0.02system 0:21.57elapsed 0%CPU (0avgtext+0avgdata 14024maxresident)k
0inputs+0outputs (0major+1119minor)pagefaults 0swaps

Expected Results: Almost instant, around 1 second:

Storage Driver: aufs
[...]
0.00user 0.04system 0:00.89elapsed 5%CPU (0avgtext+0avgdata 13496maxresident)k
0inputs+0outputs (0major+993minor)pagefaults 0swaps

Additional info:

Device	Storage driver	options	time
SSD	aufs		<1s
HDD	aufs		3s
SSD	btrfs	-s btrfs	8s
SSD	devicemapper	-s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard	14s
HDD	devicemapper	-s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard --storage-opt dm.datadev=/dev/vg0/data --storage-opt dm.metadatadev=/dev/vg0/metadata	15s
SSD	devicemapper	-s devicemapper	22s

I tried upgrading my kernel from 3.16.7 to 3.18.0 and a fresh Docker install became orders of magnitude slower. Turns out that in Debian’s 3.18.0 kernel AUFS is not enabled, and it uses devicemapper backend, whereas in 3.16.7 there is AUFS. This might also explain why docker seemed so slow on the production servers (running CentOS7), and used to be reasonably fast on my desktop (Debian Jessie).

With 3.16.7-ckt2-1 and aufs it takes under 1 second to run the testcase, but with devicemapper it takes almost half a minute. Even running aufs on a HDD is faster than running devicemapper/btrfs on an SSD!

Everytime when switching between storage drivers I wiped /var/lib/docker (sometimes also had to manually unmount, dmsetup clear, dmsetup remove when containers got stuck):

$ sudo service docker stop
$ sudo rm -rf /var/lib/docker
$ sudo mkdir /var/lib/docker
... set DOCKEROPTS="-s devicemapper"` in /etc/default/docker
$ sudo service docker start

Now /var/lib/docker is on an SSD with XFS, and I noticed that docker by default uses discard which is well known to be slow.

With discard disabled on 3.16.7 it is still slow:

0.00user 0.03system 0:13.72elapsed 0%CPU (0avgtext+0avgdata 13492maxresident)k
0inputs+0outputs (0major+1015minor)pagefaults 0swaps

Also tried btrfs storage backend, or using a direct lvm on a HDD instead of a SSD to make sure that discard really has no impact, with similar results. Direct LVM on the HDD with 3.18.0:

0.06user 0.07system 0:14.54elapsed 0%CPU (0avgtext+0avgdata 14004maxresident)k
0inputs+0outputs (0major+1145minor)pagefaults 0swaps
$ sudo docker info
Containers: 16
Images: 19
Storage Driver: devicemapper
 Pool Name: docker-8:18-269820673-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 480.4 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 1.794 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.18.0-trunk-amd64
Operating System: Debian GNU/Linux 8 (jessie)
WARNING: No memory limit support
WARNING: No swap limit support
$ sudo docker version
Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): d344625
OS/Arch (client): linux/amd64
Server version: 1.3.3
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): d344625

btrfs-on-SSD:

0.02user 0.04system 0:07.34elapsed 0%CPU (0avgtext+0avgdata 13836maxresident)k
0inputs+0outputs (0major+1095minor)pagefaults 0swaps

aufs-on-HDD:

0.01user 0.04system 0:02.39elapsed 2%CPU (0avgtext+0avgdata 13936maxresident)k
0inputs+0outputs (0major+1109minor)pagefaults 0swaps

Edit: added btrs-on-SSD numbers, aufs-on-HDD numbers and checked all timings again.

Apparently AUFS is the only storage backend that is usable in a production environment with adequate performance, except that is the only place where it is not available …