moby: Base Device UUID verification failed in case of lvm resize

I’ve done a resizing of lvm partitions on centos 7 minimal install with default settings, with these commands :

umount /home/ lvremove /dev/centos/home lvextend --size +50.00GB -r /dev/mapper/centos-root lsblk sed -i ‘/centos-home/d’ /etc/fstab

and after a reboot, docker fails to start returning this error:

prior storage driver “devicemapper” failed: Base Device UUID verification failed

CLARIFICATION: Docker data was stored on default installation path ** /var/lib/docker** mounted on ** /dev/mapper/centos-root**. The deleted lvm ** /dev/centos/home** was totally empty.

I think this is because of the new functionality introduced with docker 1.8 : https://github.com/docker/docker/issues/13896

It should be there a way to reinizialize the uuid, because actually i’m with the system totally broken.

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 57 (22 by maintainers)

Most upvoted comments

Got my data back by just changing the UUID here:

[root@docker-host-fkb docker]# grep "aae8eb6b-2998-4018-9531-bbbdc12d46b8" * -R
devicemapper/metadata/deviceset-metadata:{"next_device_id":1,"BaseDeviceUUID":"aae8eb6b-2998-4018-9531-bbbdc12d46b8","BaseDeviceFilesystem":"xfs"}

stefanleh on Mar 22, 2016

Getting the same issue after having run out of disk space on a centos VM. Is there a way to recover my docker data?

astewart-twist on Nov 20, 2015

@gordontyler So here is the summary of what I have tried.

Problem happens due to thin pool being full. Now pool has reached a state where it needs to go through thin_check/thin_repair.
For the case of loop devices, I grew the space fs under loop devices, shutdown thin pool, used thin_repair to repair metadata file and replaced old metadata file with new repaired metadata file and restarted docker and that got me going.
It is recommended to use lvm thin pool (instead of using thin pool on top of loop devices). In that case you should be simple be able to add more space to volume group backing up lvm thin pool and then I just had to deactivate the lvm thin pool and reactivate it again and it worked for me.

rhvgoyal on Nov 6, 2015

Ok, lets try to do some debugging. First of all make sure you have copied all your data. So that if we lose something in debugging process, you can restore it.

Try following. And see if you can activate thin pool and base thin device and if UUID matches the one stored in docker.

Stop docker

$systemctl stop docker
Run “losetup” and make sure /var/lib/docker/devicemapper/devicemapper/metadata is mapped to /dev/loop0 and /var/lib/docker/devicemapper/devicemapper/data is mapped to /dev/loop1. Otherwise run following.

$ losetup -f data $ losetup -f metadata
Create thin pool.

$ dmsetup create thin-pool --table “0 209715200 thin-pool /dev/loop1 /dev/loop0 128 0”
Make sure pool got created

$ dmsetup table thin-pool
Find the thin id of base device and size of device.

$ cat /var/lib/docker/devicemapper/metadata/base

Look for field “device_id”, that’s the thin device id of base device. Make a note of it. Also not “size”. Say

base_device_id=X base_size=Y

Device base device size by 512 to convert it into sectors.

base_size_sectors=Y/512
Activate base thin device

$ dmsetup create base-thin --table “0 $base_size_sectors thin /dev/mapper/thin-pool $base_device_id”

ex.

$ dmsetup create base-thin --table “0 209715200 thin /dev/mapper/thin-pool 1”
Make sure thin device got activate.

$ dmsetup table base-thin
Run “blkid” and figure out UUID of base device.

$ blkid /dev/mapper/base-thin

Make a note of UUID.
Now open /var/lib/docker/devicemapper/metadata/deviceset-metadata and make sure BaseDeviceUUID stored there matches with UUID noted in previous step.

rhvgoyal on Aug 20, 2015