podman: Pull fails with "unable to find '...' in local storage: no such image"

[//]: # kind bug

Description

Podman fails to pull quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1.

Steps to reproduce the issue:

Get a pull secret from here.

$ podman --log-level debug pull --authfile pull-secret.json quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1

Describe the results you received:

DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/openshift-release-dev/ocp-v4.0:none" 
Trying to pull quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=openshift-release-dev%2Btrking1c2pjyezbotv6cy4sg7txgrvjbr&scope=repository%3Aopenshift-release-dev%2Focp-v4.0%3Apull&service=quay.io 
DEBU[0001] Increasing token expiration to: 60 seconds   
DEBU[0001] GET https://quay.io/v2/openshift-release-dev/ocp-v4.0/manifests/sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1 
DEBU[0001] IsRunningImageAllowed for image docker:quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1 
DEBU[0001]  Using default policy section                
DEBU[0001]  Requirement 0: allowed                      
DEBU[0001] Overall: allowed                             
Getting image source signatures
DEBU[0001] Manifest has MIME type application/vnd.docker.distribution.manifest.v1+prettyjws, ordered candidate list [application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0001] ... will first try using the original manifest unmodified 
Skipping fetch of repeat blob sha256:a02a4930cb5d36f3290eb84f4bfa30668ef2e9fe3a1fb73ec015fc58b9958b17
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:9afeca73426ec14b331e7968e69e8514225c232ecf782a31c08935c6eebd063c
Skipping fetch of repeat blob sha256:cc5e2e213f55098e4f22ba855286cea38f072b2eaf8c13001425dec764b0b5fd
Writing manifest to image destination
Storing signatures
DEBU[0001] setting image creation date to 2018-12-11 22:41:48.885389906 +0000 UTC 
DEBU[0001] reusing image ID "4c0e17d6c6a1d81b883702ec1ffedca65db133980465ef4c8d1279a15889a809" 
DEBU[0001] set names of image "4c0e17d6c6a1d81b883702ec1ffedca65db133980465ef4c8d1279a15889a809" to [quay.io/openshift-release-dev/ocp-v4.0:none quay.io/openshift-release-dev/ocp-v4.0:none] 
DEBU[0001] saved image metadata "{}"                    
DEBU[0001] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1" 
DEBU[0001] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1" does not resolve to an image ID 
ERRO[0001] error pulling image "quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1": error retrieving local image after pulling quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1: unable to find 'quay.io/openshift-release-dev/ocp-v4.0@sha256:535875db8b4bcb4e04a4fb1f3fc010f515681f6b34172ebb598c9d82b3ed13b1' in local storage: no such image 

Describe the results you expected:

A successful pull.

Output of podman version:

$ podman version
Version:       0.12.2-dev
Go Version:    go1.10.3
Git Commit:    "c8eaf59d5f4bec249db8134c6a9fcfbcac792519"
Built:         Wed Dec 19 01:07:06 2018
OS/Arch:       linux/amd64

Output of podman info:

$ podman info
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-0.10.1.3-1.gitdb08685.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 0e7abf050824203d775a20c26ed14f716f7d1aa7-dirty'
  Distribution:
    distribution: '"rhel"'
    version: "7.5"
  MemFree: 2805551104
  MemTotal: 16289890304
  OCIRuntime:
    package: runc-1.0.0-52.dev.git70ca035.el7_5.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 8172576768
  SwapTotal: 8254386176
  arch: amd64
  cpus: 8
  hostname: trking.remote.csb
  kernel: 3.10.0-891.el7.x86_64
  os: linux
  rootless: false
  uptime: 157h 47m 10.3s (Approximately 6.54 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
store:
  ContainerStore:
    number: 77
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 22
  RunRoot: /var/run/containers/storage

I’m not sure what “does not resolve to an image ID” is about, maybe that’s just “it’s not in storage yet, but we’re about to put it in”. There’s some discussion of the log-level for that message in containers/image#393.