podman: podman image digest showing incorrect value in some cases

/kind bug

Description

podman inspect sometimes shows different digest than what is correct.

Steps to reproduce the issue:

Look at the 30-x86_64 tag from the fedora/fedora repo on quay.io and notice the current tag is at sha256:5bc93c7ca1c526b2a73f7c97eae15638f40dfef4b44d528f4d0374302fcb9f2b.
sudo podman pull quay.io/fedora/fedora:30-x86_64
sudo podman inspect quay.io/fedora/fedora:30-x86_64 | jq '.[]["Digest"]'

Describe the results you received:

$ sudo podman inspect quay.io/fedora/fedora:30-x86_64 | jq '.[]["Digest"]'
"sha256:9a1bdd9da2e0723b1a4f6f016311c3228c0fadb867ef6fdd9496ab665c1d5130"

Describe the results you expected:

$ sudo podman inspect quay.io/fedora/fedora:30-x86_64 | jq '.[]["Digest"]'
"sha256:5bc93c7ca1c526b2a73f7c97eae15638f40dfef4b44d528f4d0374302fcb9f2b"

Additional information you deem important (e.g. issue happens only occasionally):

This is the weird part. I’ve noticed this is not consistent. For example:

My Home Fedora 29 Workstation machine (podman-1.4.4-4.fc29.x86_64) DOES have the problem
My Work Laptop Fedora 29 Silverblue Machine (`podman-1.4.4-4.fc29.x86_64) DOES NOT have the problem
My Spare Laptop Fedora 30 Siverblue Machine (podman-1.4.4-4.fc30.x86_64) DOES NOT have the problem
My coworkers Laptop Fedora 30 Silverblue Machine (podman-1.4.4-4.fc30.x86_64) DOES have the problem

Output of podman version:

$ sudo podman version
Version:            1.4.4
RemoteAPI Version:  1
Go Version:         go1.11.11
OS/Arch:            linux/amd64

Output of podman info --debug:

$ sudo podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.11.11
  podman version: 1.4.4
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 130ae2bc326106e335a44fac012005a8654a76cc'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 6809538560
  MemTotal: 20999897088
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 82f4855a8421018c9f4d74fbcf2da7f8ad1e11fa
      spec: 1.0.1-dev
  SwapFree: 4262981632
  SwapTotal: 4294963200
  arch: amd64
  cpus: 8
  hostname: media
  kernel: 5.1.21-200.fc29.x86_64
  os: linux
  rootless: false
  uptime: 52h 8m 44.94s (Approximately 2.17 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 11
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

About this issue

Original URL
State: open
Created 5 years ago
Comments: 71 (33 by maintainers)

Most upvoted comments

I am facing a similar issue with incorrect digests. On my fedora workstation fc34, digests listed via podman images --digests do not match up with the digests shown on hub.docker.com . However when I inspect the image explicitly (using podman inspect) then I get 2 RepoDigests in the json: one of the digests is the one that is shown on hub.docker.com and the other one is the digest that is shown when I do podman images --digests.

Is this expected behavior? Is it normal to have two RepoDigests ? Shouldn’t the digest shown on hub.docker.com match up with podman images --digests?

Here is an example:

$ podman images --digests
REPOSITORY                     TAG         DIGEST                                                                   IMAGE ID      CREATED       SIZE
...etc...
docker.io/library/debian       latest      sha256:dcb20da8d9d73c9dab5059668852555c171d40cdec297da845da9c929b70e0b1  7a4951775d15  4 weeks ago   119 MB
...etc...

$ podman inspect 7a4951775d15
[                  
    {                        
        "Id": "7a4951775d157843b47250a2a5cc7b561d2abe0b29ae6f19737a04635302eacf",
        "Digest": "sha256:dcb20da8d9d73c9dab5059668852555c171d40cdec297da845da9c929b70e0b1",
        "RepoTags": [
            "docker.io/library/debian:latest"
        ],             
        "RepoDigests": [  
            "docker.io/library/debian@sha256:5625c115ad881f19967a9b66416f8d40710bb307ad607d037f8ad8289260f75f",
            "docker.io/library/debian@sha256:dcb20da8d9d73c9dab5059668852555c171d40cdec297da845da9c929b70e0b1"
        ],          
...

Summary So here sha256:dcb20da8d9d73c9dab5059668852555c171d40cdec297da845da9c929b70e0b1 shown via podman images --digests does NOT match https://hub.docker.com/layers/debian/library/debian/latest/images/sha256-5625c115ad881f19967a9b66416f8d40710bb307ad607d037f8ad8289260f75f?context=explore

But interestingly the sha256 on hub.docker.com DOES appear in the RepoDigests array in the json (as the first entry).

$ podman --version
podman version 3.2.2

I’m experiencing this with simple popular images from hub.docker.com like hello-world, ubuntu, debian.

sidkshatriya on Jul 21, 2021

@SarthakGhosh16 That seems reasonable at a first glance; podman build creates an uncompressed representation of the image, podman push writes a compressed one. The two are going to have different digests.

This message gave me a glimpse of hope. So I tried podman build --disable-compression=false, but I still get a different hash compared to the one showing up on ghcr.io.

Is there any way to get the hash of what podman pushes, ugly or not? Because I need to build an image locally and push it to ghcr.io and then be able to obtain the digest locally so I can sign it and give to people. The people pulling down the image should not need to trust ghcr.io, but rather only need to trust me and my digest that the image is authentic.

EDIT: adding --digestfile=save-here-plix to podman push will store the digest of what was pushed in save-here-plix 🥳

faern on Oct 21, 2022

Still getting the issue

Frizzay on Jan 28, 2021