podman: Error reading image ... image not known

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Several times since I switched to podman from docker I end up getting an image not known error. I’m unable to remove the offending image and the only solution I’ve found (besides manually combing through the storage and json files to clean it out) is to wipe ~/.local/share/containers/storage and start over.

I typically first notice the error when running podman image:

$ podman image ls
ERRO[0003] error checking if image is a parent "7030953edf669a9fcdd40594d588ea7a6af389c65f6231ee1126066267605963": error reading image "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": image not known 
ERRO[0003] error checking if image is a parent "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": error reading image "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": image not known 
REPOSITORY                                                       TAG         IMAGE ID       CREATED          SIZE
<none>                                                           <none>      e0052b5acc35   5 minutes ago    568 MB
<none>                                                           <none>      4e13c4def0df   10 minutes ago   568 MB
<none>                                                           <none>      7c5da149659b   16 minutes ago   568 MB
localhost/seed-loopback3                                         latest      fa7398dcd3cb   17 hours ago     632 MB
<none>                                                           <none>      94ef4d389a1e   18 hours ago     699 MB
<none>                                                           <none>      f0c5b4e82e11   3 days ago       699 MB
<none>                                                           <none>      7030953edf66   3 days ago       699 MB
<none>                                                           <none>      493974907437   3 days ago       699 MB
localhost/impersonator                                           latest      556e8499d9f9   4 days ago       930 MB
<none>                                                           <none>      c0bcb0376d5a   4 days ago       692 MB
<none>                                                           <none>      b5769e0bd61a   4 days ago       692 MB
registry.example.org/containers/base-images/rhel7-node-oracle   8           fe5c0145a44a   4 days ago       793 MB
registry.redhat.io/rhoar-nodejs/nodejs-10                        latest      0d01232685c9   5 days ago       575 MB
registry.example.org/containers/redhat/nodejs-8                 latest      36bd7e8bcf55   5 days ago       568 MB
registry.access.redhat.com/rhoar-nodejs/nodejs-8                 latest      36bd7e8bcf55   5 days ago       568 MB
registry.example.org/containers/base-images/rhel7-mysql-80      latest      815d76a89459   13 days ago      478 MB
docker.io/bmaupin/pspdev                                         gcc-4.6.4   cedbe259e0a0   2 months ago     911 MB

I can’t remove the offending image:

$ podman image rm f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21
Error: error reading image "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": image not known

$ podman image rm -f f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21
Error: error reading image "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": image not known

I can’t do a prune either:

$ podman image prune -a
Error: failed to prune image: Image used by f2061acd5a510ad39a7ec7923d2a1aa416210ef4c2cd5c28afec92d6c4a677a1: image is in use by a container

Where does f2061acd5a510ad39a7ec7923d2a1aa416210ef4c2cd5c28afec92d6c4a677a1 come from? It’s not in the list of images or containers, and I can’t remove it either.

$ podman container ls -a
CONTAINER ID  IMAGE                                                               COMMAND               CREATED       STATUS                     PORTS                   NAMES
131af825d0cc  localhost/seed-loopback3:latest                                     node_modules/.bin...  17 hours ago  Exited (127) 17 hours ago                          seed-loopback3
815ebf0b8933  localhost/impersonator:latest                                       node_modules/.bin...  4 days ago    Exited (130) 4 days ago                            impersonator
348fd256a08b  docker.io/bmaupin/pspdev:gcc-4.6.4                                  ./libretro-buildb...  5 days ago    Created                                            thirsty_moore
a5b76fd2b773  registry.example.org/containers/base-images/rhel7-mysql-80:latest  sh -c run-mysqld ...  11 days ago   Exited (0) 5 days ago      0.0.0.0:3306->3306/tcp  mysql

$ podman image rm f2061acd5a510ad39a7ec7923d2a1aa416210ef4c2cd5c28afec92d6c4a677a1
Error: unable to find a name and tag match for f2061acd5a510ad39a7ec7923d2a1aa416210ef4c2cd5c28afec92d6c4a677a1 in repotags: no such image

$ podman container rm f2061acd5a510ad39a7ec7923d2a1aa416210ef4c2cd5c28afec92d6c4a677a1
Error: no container with name or ID f2061acd5a510ad39a7ec7923d2a1aa416210ef4c2cd5c28afec92d6c4a677a1 found: no such container

Steps to reproduce the issue:

Although this has happened several times, I’m not sure what the cause is and so I’m not sure how to reproduce it.

Describe the results you received:

See Description

Describe the results you expected:

See Description

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 3998068736
  MemTotal: 16756355072
  OCIRuntime:
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 8161837056
  SwapTotal: 8189374464
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: host
  kernel: 5.0.0-27-generic
  os: linux
  rootless: true
  uptime: 88h 40m 44.79s (Approximately 3.67 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
store:
  ConfigFile: /home/user/.config/containers/storage.conf
  ContainerStore:
    number: 17
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/user/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 65
  RunRoot: /tmp/1000
  VolumePath: /home/user/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/bionic,now 1.5.1-1~ubuntu18.04~ppa1 amd64 [installed]

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical machine running Ubuntu 18.04.

Thanks!