distribution: Random "image config verification failed for digest sha256:" errors when pulling images
Output of docker version:
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:54:52 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:54:52 2016
OS/Arch: linux/amd64
Output of docker info:
Containers: 165
Running: 147
Paused: 0
Stopped: 18
Images: 293
Server Version: 1.10.3
Storage Driver: btrfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.2.0-34-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 31.42 GiB
Name: <edited>
ID: AZNH:RYE6:EPU2:OM3H:RQGL:E6GL:3PCA:BJJP:JVQO:ILYS:G7B4:TWOZ
Username: <edited>
Registry: https://index.docker.io/v1/
Additional environment details (AWS, VirtualBox, physical, etc.):
VPS
** Registry setup **
We’re running our own distribution registry, version 2.3 backed by a Swift storage backend and with a nginx reverse-proxy on two instances of the registry. (with redis cache)
Steps to reproduce the issue:
- docker run <image> <cmd>
- docker commit <id> / docker tag
- docker push image:tag
Note: The layers pushed are not very large, maximum 300-400MB
On another host
- docker pull image:tag
Describe the results you received:
└> sudo docker pull repo:e88527622a7d52a
[sudo] password for leo:
e88527622a7d52a: Pulling from repo
a3ed95caeb02: Already exists
4546a4a8a29d: Already exists
a898f8c7f6ba: Already exists
706514fbad74: Already exists
979168d7ceb7: Already exists
6a953df111c8: Already exists
39cb19057166: Already exists
0d268101d605: Already exists
517ff1a63fec: Already exists
cc017fff06a9: Pulling fs layer
image config verification failed for digest sha256:9315cd9d987b2ec50be5529c18efaa64ebae17f72ae4b292525ca7ee2ab98318
Describe the results you expected:
No error
** Notes **
It happens one image among hundreds, more or less, it happens almost everyday in our production infrastructure but I can’t reproduce it willingly.
What can be the cause of that ?
About this issue
- Original URL
- State: open
- Created 8 years ago
- Comments: 20 (5 by maintainers)
Commits related to this issue
- don't swallow errors in Swift driver's GetContent() In 326c3a9c49486f4c5dbb16508f08b6b766bd4852, which was only intended to be a refactoring commit, the behavior of this block subtly changed so that ... — committed to distribution/distribution by majewsky 8 years ago
- don't swallow errors in Swift driver's GetContent() In 326c3a9c49486f4c5dbb16508f08b6b766bd4852, which was only intended to be a refactoring commit, the behavior of this block subtly changed so that ... — committed to dalsh/distribution by majewsky 8 years ago
- don't swallow errors in Swift driver's GetContent() In 650df321bbad2dae1ecea47735b42148ea7f3c9d, which was only intended to be a refactoring commit, the behavior of this block subtly changed so that ... — committed to thaJeztah/distribution by majewsky 8 years ago
- don't swallow errors in Swift driver's GetContent() In 000dec3c6f6e92ec20cb86d1375ec82d2f6062b3, which was only intended to be a refactoring commit, the behavior of this block subtly changed so that ... — committed to thaJeztah/distribution by majewsky 8 years ago
- don't swallow errors in Swift driver's GetContent() In 000dec3c6f6e92ec20cb86d1375ec82d2f6062b3, which was only intended to be a refactoring commit, the behavior of this block subtly changed so that ... — committed to digitalocean/docker-distribution by majewsky 8 years ago
I am facing the same issue with swift backend on OVH. I get intermittent
image config verification failed for digesterrors on pull anddigest invalid: provided digest did not match uploaded contenton push. Also, once an image upload is corrupted it is very difficult to restore the image as consecutive pushes will not rewrite the corrupt layer.Related issue with the swift backend: https://github.com/docker/distribution/issues/1013