distribution: Random "image config verification failed for digest sha256:" errors when pulling images

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 165
 Running: 147
 Paused: 0
 Stopped: 18
Images: 293
Server Version: 1.10.3
Storage Driver: btrfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.2.0-34-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 31.42 GiB
Name: <edited>
ID: AZNH:RYE6:EPU2:OM3H:RQGL:E6GL:3PCA:BJJP:JVQO:ILYS:G7B4:TWOZ
Username: <edited>
Registry: https://index.docker.io/v1/

Additional environment details (AWS, VirtualBox, physical, etc.):

VPS

** Registry setup **

We’re running our own distribution registry, version 2.3 backed by a Swift storage backend and with a nginx reverse-proxy on two instances of the registry. (with redis cache)

Steps to reproduce the issue:

  1. docker run <image> <cmd>
  2. docker commit <id> / docker tag
  3. docker push image:tag

Note: The layers pushed are not very large, maximum 300-400MB

On another host

  1. docker pull image:tag

Describe the results you received:

└> sudo docker pull repo:e88527622a7d52a
[sudo] password for leo: 
 e88527622a7d52a: Pulling from repo

a3ed95caeb02: Already exists 
4546a4a8a29d: Already exists 
a898f8c7f6ba: Already exists 
706514fbad74: Already exists 
979168d7ceb7: Already exists 
6a953df111c8: Already exists 
39cb19057166: Already exists 
0d268101d605: Already exists 
517ff1a63fec: Already exists 
cc017fff06a9: Pulling fs layer 
image config verification failed for digest sha256:9315cd9d987b2ec50be5529c18efaa64ebae17f72ae4b292525ca7ee2ab98318

Describe the results you expected:

No error

** Notes **

It happens one image among hundreds, more or less, it happens almost everyday in our production infrastructure but I can’t reproduce it willingly.

What can be the cause of that ?

About this issue

  • Original URL
  • State: open
  • Created 8 years ago
  • Comments: 20 (5 by maintainers)

Commits related to this issue

Most upvoted comments

I am facing the same issue with swift backend on OVH. I get intermittent image config verification failed for digest errors on pull and digest invalid: provided digest did not match uploaded content on push. Also, once an image upload is corrupted it is very difficult to restore the image as consecutive pushes will not rewrite the corrupt layer.

Related issue with the swift backend: https://github.com/docker/distribution/issues/1013