podman: "lstat: invalid argument" when using 'overlay' storage driver with fuse-overlayfs

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman in rootless mode with the overlay (or overlay2, they are the same from what I can tell) storage driver (with fuse-overlayfs) fails.

Steps to reproduce the issue:

1. Run podman container run --rm -it hello-world

Describe the results you received:

Running anything with overlayfs on Gentoo causes the following error:

Error: error resolving storage path for container [id]: lstat /home/[user]/.local/share/containers/storage/overlay2/[id]/merged: invalid argument

I modified container_internal.go to exit as soon as it hits that error (to prevent any cleanup), and the container [id] directory ends up containing only the following items:

$ find .
./merged
./work
./work/work
./lower
./link
./diff

in storage driver directory: (note the hello binary being present)

rafael@RPC ~/.local/share/containers/storage/overlay2 $ find .
.
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/empty
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/merged
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/work
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/link
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/diff
./9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63/diff/hello
./l
./l/F7MDBH2YNJJOT3IHFYIKL3G4OW
./l/AFHTK4I5REAE6ZYPU6QKDFHYZO
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/merged
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/work
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/work/work
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/lower
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/link
./37fd79dd6c7c9919b12660c74ffe4312fd36b5bda10448f1f1d3a5535504a021/diff

Additional information you deem important (e.g. issue happens only occasionally):

vfs storage driver works just fine

Output of podman version:

podman version 2.2.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.22, commit: 9c34a8663b85e479e0c083801e89a2b2835228ed'
  cpus: 4
  distribution:
    distribution: gentoo
    version: unknown
  eventLogger: file
  hostname: RPC
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1065536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1065536
      size: 65536
  kernel: 5.5.0RMOD-00003-g35c07c7e8a29
  linkmode: dynamic
  memFree: 786419712
  memTotal: 8217759744
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.3.0
  swapFree: 0
  swapTotal: 0
  uptime: 117h 4m 17.82s (Approximately 4.88 days)
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: mirror.gcr.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/rafael/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay2.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/rafael/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /tmp/containers-user-1000/containers
  volumePath: /home/rafael/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1610749743
  BuiltTime: Fri Jan 15 23:29:03 2021
  GitCommit: d8868746fd3f45b9bfd79e37bf2aa0c7103245fd
  GoVersion: go1.15.6
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

app-emulation/libpod-2.2.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical Gentoo Linux

storage.conf:

[storage]
  driver = "overlay2"
  graphroot = "/home/rafael/.local/share/containers/storage"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"
additionalimagestores = [
]