moby: Docker Engine fails to restart - Base Device UUID and Filesystem verification failed
Output of docker version:
Docker version 1.11.1-cs2, build 7cd1c0b
Output of docker info:
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Additional environment details (AWS, VirtualBox, physical, etc.):
Physical environment
Steps to reproduce the issue:
- Docker was installed without docker_device config
- Used Contiv Cluster-Manager to make the host part of contiv cluster manager, which uses ‘device_mapper’ functionality.
- Re-start of Docker Engine fails with above error message and ansible play-book ran for node commissioning to the contiv-cluster fails.
Describe the results you received: Docker Engine fails to restart with below error -
May 29 07:16:42 contiv-aci-scale-1.cisco.com systemd[1]: Starting Docker Application Container Engine… May 29 07:16:43 contiv-aci-scale-1.cisco.com docker[32236]: time=“2016-05-29T07:16:43.000531244-07:00” level=info msg=“New containerd process, pid: 32241\n” May 29 07:16:44 contiv-aci-scale-1.cisco.com docker[32236]: time=“2016-05-29T07:16:44.018947601-07:00” level=error msg=“[graphdriver] prior storage driver "devicemapper" failed: devmapper: Base Device UUID and Filesystem verification failed.devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed” May 29 07:16:44 contiv-aci-scale-1.cisco.com docker[32236]: time=“2016-05-29T07:16:44.019375151-07:00” level=fatal msg=“Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed.devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed” May 29 07:16:44 contiv-aci-scale-1.cisco.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE May 29 07:16:44 contiv-aci-scale-1.cisco.com docker[32236]: time=“2016-05-29T07:16:44-07:00” level=info msg=“stopping containerd after receiving terminated” May 29 07:16:44 contiv-aci-scale-1.cisco.com systemd[1]: Failed to start Docker Application Container Engine. May 29 07:16:44 contiv-aci-scale-1.cisco.com systemd[1]: Unit docker.service entered failed state. May 29 07:16:44 contiv-aci-scale-1.cisco.com systemd[1]: docker.service failed.
Describe the results you expected: Docker Engine should start gracefully, when its get configured with Docker_Device functionality.
Additional information you deem important (e.g. issue happens only occasionally): Consistently re-produceable failure.
Work around is manual - Need to clean-up /var/lib/docker and reboot the node. Re-issuing cluster-commissioning task this time gets through and Docker Engine with ‘docker_device’ config this time does get successfully started.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 7
- Comments: 25 (10 by maintainers)
I found this problem after a yum update in Red Hat. Delete (or move) devicemapper folder and start docker, it will be recreated and docker will start normally:
$ rm -rf /var/lib/docker/devicemapper $ systemctl start docker
** # rm -rf /var/lib/docker ** # reboot
And re-commissioning the node for contiv cluster fixes the issue and docker engine does gets started with docker_device configs.
workaround is - rm -rf /var/lib/docker & restart docker(sometimes host reboot needed)
@rtnpro : I face your probem but i issue command:
dmsetup udevcomplete_alland pressyto confirm.And it work like a charm.
I think this is very annoying, I have this on a raspberry pi3 with aarch64 openSuSe. Moved the whole /var/lib/docker to a different btrfs subvol, because the /var/lib/docker/devicemapper/devicemapper contains a 2GB and a 100GB sparsefile, which explodes my rsync-backup of the root partion… and then I get this UUID mismatch. Again, simple deleting everything in the new partition (rather than re-using the moved-over contents) helped. Fortunately, I had to rebuild only the one container I had just made and not yet really done anything… I would have appreciated if I could move dockerdata around to whatever phys. medium or network or cluster fs without that hassle… On the other hand, it’s rather the distros shortcoming that there seems to be no simple way to get AUFS installed on suse LEAP, and that is the only running aarch64 distro for the pi3 right now afaik.