buildah: Building containers fail with error "error locating item named "manifest" for image with ID"

Description

In Openstack TripleO jobs we build a lot of containers, and sometimes it fails with error after a few tens of containers were built already:

error locating item named "manifest" for image with ID "....": file does not exist

For example:

'error checking if cached image exists from a previous build: error getting history of 
"68c139527f8459ba7c981a11f1872a6bb5b621c2140a6f2fa7858214df8e429f": error creating new image from reference to image
 "68c139527f8459ba7c981a11f1872a6bb5b621c2140a6f2fa7858214df8e429f": error locating item named "manifest" for image 
with ID "68c139527f8459ba7c981a11f1872a6bb5b621c2140a6f2fa7858214df8e429f": file does not exist\n'

Steps to reproduce the issue:

1. It happens sporadically on CI jobs in Openstack TripleO, tracked in launchpad bug: “Building container fails because manifest is not found” Command to build can be found in log: Command: sudo buildah bud --volume /etc/yum.repos.d:/etc/yum.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/yum.repos.d:/etc/yum.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --format docker --tls-verify=False --layers --logfile /home/zuul/container-builds/7d364ba7-6c15-4676-98c7-813f36f44452/base/ovn-base/ovn-base-build.log -t 127.0.0.1:5001/tripleomaster/openstack-ovn-base:a9a790d0723c9fe6641e453c6a1f0c91 /home/zuul/container-builds/7d364ba7-6c15-4676-98c7-813f36f44452/base/ovn-base

Describe the results you received: After we build tens of containers it may fail, doesn’t happen all the time. Seems like a race condition.

Describe the results you expected: To build all hundred containers.

Output of rpm -q buildah or apt list buildah:

buildah.x86_64                                1.11.6-7.module_el8.2.0+458+dab581ed           centos-appstreams  

Output of cat /etc/*release:

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Output of uname -a:

kernel-4.18.0-193.19.1.el8_2.x86_64

Output of cat /etc/containers/storage.conf:

driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
size = ""
override_kernel_check = "true"
[storage.options.thinpool]
ostree_repo = ""
skip_mount_home = "false"