podman: Podman save / load across machines causes cache misses and rebuilding of images

/kind bug

Description

I’m trying to save an image (with layers) on one machine and then load it later on another in order to accelerate our CI build pipeline. However, after loading I’m still getting a cache miss even at the first line of my build file. When I re-run my build locally, podman build get’s full cache hits and my builds are super fast.

This is a feature that seems to be supported by docker - I roughly followed this guide to do this (obviously replacing docker for podman).

Steps to reproduce the issue:

1. podman save image
2. Switch to another machine
3. podman load image.tar
4. podman build

Describe the results you received:

All instructions in the build file are run.

Describe the results you expected:

Instructions that have not changes should be able to use the cached layers from podman save

Additional information you deem important (e.g. issue happens only occasionally):

I think this is a supported behaviour with docker - I asked on the #crio slack channel and @mheon suggested this is a bug in buildah/podman and that I should file a bug.

I have tried to narrow my dockerignore file to include only the files my build absolutely needs, but even my first FROM blah and RUN commands are getting cache misses before any COPY command.

Potentially related bug: https://github.com/containers/buildah/issues/2954

Output of podman version:

3.4.2

Output of podman info:

NA

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No - using ubuntu so 3.4.2 is the latest.

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on MS hosted build agents on Azure Devops.

Keywords:

Podman build load save layers