distribution: Trying to pull : layers from manifest don't match image configuration
Hello,
I have two server, one pulling & running containers, one hosting the private registry, both running debian & docker 1.10 The private registry running 2.3, with TLS and an NFS storage.
When I want to pull I have this issue : “layers from manifest don’t match image configuration”
My docker info from the one running containers :
Containers: 2 Running: 1 Paused: 0 Stopped: 1 Images: 132 Server Version: 1.10.0 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 179 Dirperm1 Supported: true Execution Driver: native-0.2 Logging Driver: json-file Plugins: Volume: local Network: bridge null host Kernel Version: 3.16.0-4-amd64 Operating System: Debian GNU/Linux 8 (jessie) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 31.29 GiB Name: worker0 ID: NQKS:BWR4:SQKA:46OP:WZHK:5KNF:CIGX:OPM6:CV3N:QYYM:HNV5:S2WB WARNING: No memory limit support WARNING: No swap limit support WARNING: No oom kill disable support WARNING: No cpu cfs quota support WARNING: No cpu cfs period support
Any clue ? Should I just remove the /var/lib/docker/aufs/layers/ folder ?
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 55 (6 by maintainers)
Same here with docker hub! since today
The existing layers may be an important piece of the puzzle. I think what’s happening is that one of the existing layers was downloaded by an older version of Docker that did not save tar-split metadata. The migration to content-addressability computed an ID for the layer, but this doesn’t match the ID that comes directly from the tar. The migration also prepopulated the mapping between the layer digest and the artifact digest in
/var/lib/docker/image/*/distribution. Pulling another image that uses the affected layer will try to use this layer digest from the migration, but that digest turns out to be wrong, so the pull fails.If this is correct, deleting
/var/lib/docker/image/*/distributionshould fix your problem pullingmaxpowa/npomf:latest. This is just a cache, so it should be safe to remove (removing it will just cause extra data to be transfered during pushes and pulls until the cache is repopulated).The
cibox/mysqlissue is not quite the same, but could be related. That is a corrupted image on the registry, so anyone who tries to pull it will have problems, regardless of their local layers. Even with the safeguards we introduced in Docker 1.10.1, if the layer involved is not actually pushed up to the registry (because the registry already had that layer), the image config could contain the faulty layer ID from the migration process.cc @tonistiigi
I had the same issue with an image built by docker hub when trying to pull it to a server (that has been running older versions of the same image, on multiple different versions of docker, for multiple years).
@aaronlehmann’s solution of removing
/var/lib/docker/image/*/distributionresolved the issue.Hi, So can we come with a solution, even dirty ? What I will try :
What do you think ?
I had the same issue with Docker hub and after reading all the comments this fixed it for me:
Done.
@alexamirante what FROM image are you using? The problem occurs when you have mixed manifest versions, so if your own image is not the culprit, then your base image (or its own base image) is.