distribution: Local images ID does not match registry manifest digest

I will try and answer this question as best as I can and hopefully can get a better understanding of where we are missing documentation. Most of these answers only apply in docker 1.10+ and docker registry 2.3+ since those are the releases in which we made the image identifier content addressable and introduced schema 2 of the manifest used by the registry. When using older versions of docker or the registry the answer may require pulling an image to get the identifier used within docker.

What is the relationship between the sha256 hash provided by docker images or docker inspect on the one hand, and the one provided by docker pull or docker_content_digest header on the other?

The digest used for docker pull represents the digest of image manifest which is stored in a registry. This digest is considered the root of a hash chain since the manifest itself contains the hash of the content which will be downloaded and imported into docker. See the schema 2 spec for a description of this manifest https://docs.docker.com/registry/spec/manifest-v2-2/. The image id used within docker can be found in this manifest as config.digest. This config represents the image configuration which will be used within docker. So you could say the manifest is the envelope and image is what is inside. The manifest digest will always be different than the image id BUT for any given manifest the same image id should always be produced. Since it is a hash chain, we cannot guarantee that the manifest digest will always be the same for a given image id. In most cases it should usually produce the same digest, we just cannot guarantee it but do a best effort. The possible difference in manifest digest is because we do not store gzipped blobs locally and exporting of layers may produce a different digest, even though the uncompressed content should remain the same. The image id itself verifies that uncompressed content is the same, this is what we mean when we say the image id is now a content addressable identifier.

How can I use this to determine if a local image is identical to the one in the registry? Without ripping deeply into the docker engine code, what algorithm can I follow to check?

The best way to determine this is to compare the image id within a manifest to a local image id. This does require pulling and parsing the manifest.

I also ran across a discussion ( sorry cant find it now ), in which it was said that it is intentional that the digest is not stored alongside the image when a tag is pulled.

The reason we do not store the manifest identifier is because we do not store the manifest. It does not make sense to store the manifest since we are not storing the compressed blobs referenced by the manifest. We have taken measures to try and ensure the exact manifest is pushed (cross repository push, caching compressed blob identifiers associated with each layer). We do store the manifest hash when pulling by digest, but this is only a read only value to reference the correct image id in the future without pulling and not exportable since you cannot push by digest.

If you have upgraded from an older version of the registry or docker client and do not find these schema 2 manifests to be present in your registry, then repushing images should produce the new manifest. Also a disclaimer that the schema 2 manifests are currently not enabled on the public hub registry.

I’ve been trying to work out how to do this, and hitting a lot of brick walls. Would anyone be able to share a working curl statement that returns something I can use to compare against the current image?

This is really funny. I was doing some work on manifests again, saw the hash differences between manifest and local, and thought I remembered someone opening an issue on it. Did a Google search and came across one opened by… me! 😆

This completely makes sense, and thanks for the help a few years ago @dmcgowan . I can compare the config digest and it matches to the one shown in docker image ls (or docker image inspect).

Do you mind explaining how the layers digests work? The digests of layers in the manifest are the digests of the compressed blobs, and that works. On the other hand, the digests of the layers in docker image inspect do not match up to those. Since docker doesn’t store the compressed blobs, I can see why you might not want those, but what do the hashes in docker image inspect represent? Hashes of what?

I put a bounty on this question on stack overflow and got a good answer, though I still had trouble using it for my goal. Maybe you will have more luck:

https://stackoverflow.com/questions/61366738/how-are-the-docker-image-layer-ids-derived/69688979?noredirect=1

this needs to be fixed because this has an implication on stack deploy - since local hashes are not computed properly, stack deploy actually recreates each and every service … even those that havent changed.

Yep, that’s my problem. I checked, and I have a version 1 schema still. I’ll upgrade and repush.

So I think the super-simple answer to my question is:

IF you are using a 2.3+ registry and 1.10+, then the manifest contains config.digest, which is used as the docker image id. IF NOT, then the docker image ID is something else, and will not match anything in the manifest.

On Thu, Apr 21, 2016 at 4:21 PM, Derek McGowan notifications@github.com wrote:

@dcowden https://github.com/dcowden if the images was pushed with 1.10+ onto a registry 2.3+ then the image id should be in there. If it is a schema 2 manifest (see schemaVersion key) and the config.digest differs, try pulling the image and seeing if the image id of the pulled image matches. If it not trying a re push and you should get a schema 2 manifest referencing the image id.

If you are still seeing issues would be helpful to see the output of the docker commands as well as the manifest in question.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/docker/distribution/issues/1662#issuecomment-213099290