moby: Docker Install: Error running DeviceCreate (createPool) dm_task_run failed

# kill -9 $(lsof -t -c docker)
# rm -rf /var/lib/docker/*
# reboot

Works for me.

TL;DR

I’m constantly corrupting my /var/lib/docker directory by running docker in loopback mode when I MEAN to run in devicemapper mode.

To fix without rebooting/deleting the ENTIRE /var/lib/docker dir (as root)

1. dmsetup ls # list docker-* devices
2. dmsetup remove docker-* #fill in the * yourself, unmaps loopback device
3. rm -rv /var/lib/docker/devicemapper/devicemapper
4. Restart docker in devicemapper mode

Explanation

I’m not sure if this is of any help, but I am CONTINUOUSLY having this problem too!

...
DEBU[0000] devmapper: Added deviceId=149 to DeviceIdMap 
DEBU[0000] devmapper: Added deviceId=226 to DeviceIdMap 
DEBU[0000] devmapper: constructDeviceIDMap() END        
DEBU[0000] devmapper: activateDeviceIfNeeded()          
DEBU[0000] devmapper: Error device setupBaseImage: devmapper: Base Device UUID and Filesystem verification failed.devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed 
DEBU[0000] Cleaning up old shm/mqueue mounts: start.    
FATA[0000] Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed.devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed 

Now the reason I get this is VERY reproducible and fixable. (NO, NOT BY DELETING /var/lib/docker!!!)

In my case, I run docker via

/usr/bin/docker daemon -H fd:// --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G

Which I add to /usr/lib/systemd/system/docker.service, HOWEVER every time I upgrade, that file get reset, and I inevitable forget to change it (AGAIN!) before trying to start docker… And here’s the kick, when you start docker daemon without these options (in Centos 7), it will start in loopback mode or whatever. And when THAT happens, /var/lib/docker/devicemapper/devicemapper/ is populated. So when you DO run with the --storage-opt dm option, NOW the directory is corrupted. However, what I just discovered is that by deleting the /var/lib/docker/devicemapper/devicemapper/ directory ONLY, the problem is fixed. I’m able to start docker with the dm storage and keep all my images and containers…

Edit: However I have discovered that it is important to clean up the loop back devices properly BEFORE deleting /var/lib/docker/devicemapper/devicemapper/, or else you have stray loopbacks that need to be cleaned up (via rebooting?)

Recap

1. /usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G #works

2. /usr/bin/docker daemon #corrupts

3. /usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G #broken

4. dmsetup ls to list the mapped docker devices

   vg--docker-data (253:2)
   vg--docker-metadata (253:3)
   docker-253:1-2152645735-pool    (253:4)
   centos-swap (253:0)
   centos-root (253:1)
   

5. dmsetup remove docker-253:1-2152645735-pool # This should remove the map on the loopback device AND remove the loopback devices, since autoclear is usually on, the vgs are something else, and belong there

6. losetup -a #lists loopback devices (Should be empty of docker loopbacks)

7. losetup -d /dev/loop0 # to unmount loopback device IF any are left. only unmount the ones you see in losetup -a, which might not be loop0

8. rm -rv /var/lib/docker/devicemapper/devicemapper #fixes everything

9. /usr/bin/docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.basesize=100G #works again

Running CentOS 7.2.1511

docker version

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:34:50 2016
 OS/Arch:      linux/amd64

uname -a

Linux newbie 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

As a side note, the third time I tried this, I had to delete deviceset-metadata and transaction-metadata from the /var/lib/docker/devicemapper/metadata directory. This may be unrelated, but the values got messed up, and deleting them lets them regenerate safely.

this helped for me: (after switching to the latest kernel on DigitalOcean)

sudo apt-get install linux-image-extra-`uname -r`

I had the same problem and was not able to fix it because I did’nt want to go for the full nuke: /var/lib/docker/ solution. I found something promising at: http://grokbase.com/t/gg/docker-user/1563fzdtm7/docker-docker-runs-out-of-space-when-trying-to-create-a-new-image 'The default docker storage driver allocates 10GB storage blocks for your images. Move to overlayfs and avoid this entirely. In the command that starts your docker daemon just add “-s overlay” ’

Are you using digital ocean?

I was once suffering from the same problem until I run

locate “.config”

and then switched my kernel to one shown at the last command.

This worked for me, please try if this works for you as well.

I received this error when docker used up all it’s disk space. I went and removed a bunch of images to free some space up, but any time I tried to start a container, it threw the Error running DeviceCreate (createPool) dm_task_run failed error. I tried stopping docker and starting it back up, and now it won’t even start.

Going to have to wipe the docker partition and start over.

ah, need to include --storage-opt="dm.blocksize=64k" to the daemon flags for the time being