podman: Rootless Podman won't work since version 3.1.0

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

This subject was initially discussed at https://bbs.archlinux.org/viewtopic.php?id=265140.

Since v3.1.0, rootless Podman is not working with the same configuration that worked in v3.0.1 for some images.

It is hanging with zero output (only when ran in rootless mode!) with rootless_storage_path defined with “$HOME” variable. If rootless_storage_path is commented out or set to an user’s home path (instead of $HOME variable), it sort of works. It is as if $HOME was being used literally, instead of as a variable.

When I say “sort of works” it is because it still gives “operation not permitted” with some Docker images, like archlinux or ubuntu. However, I tested with alpine and fedora, and it works. See error outputs below.

Steps to reproduce the issue:

1. enable rootless_storage_path setting in /etc/containers/storage.conf with the following value

   rootless_storage_path = "$HOME/.local/share/containers/storage"
   

2. run podman pull docker.io/archlinux as normal user

3. the CLI should be hanging, so run Ctrl+C to cancel the previous command

4. now set rootless_storage_path with the home directory, say “/home/foo/”

   rootless_storage_path = "/home/foo/.local/share/containers/storage"
   

5. run podman pull docker.io/archlinux as normal user again

6. layers are download, but it ends up with “operation not permitted” error message

Describe the results you received:

Terminal is hanging when rootless_storage_path is set and has “$HOME” in its path:

$ podman pull docker.io/archlinux

If “$HOME” is replaced with real valid path like “/home/foo”, I get the following output:

$ podman pull docker.io/archlinux
Trying to pull docker.io/library/archlinux:latest...
Getting image source signatures
Copying blob 10756994dc19 skipped: already exists  
Copying blob 5bb50848eab8 done  
Copying config 3de742be92 done  
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): operation not permitted
Error: Error committing the finished image: error adding layer with blob "sha256:5bb50848eab8d3d80a48b3769ef342097f57881b1ef86826e898c43ee4dd2460": Error processing tar file(exit status 1): operation not permitted

But not all images:

$ podman pull docker.io/alpine
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob ca3cd42a7c95 done  
Copying config 49f356fa45 done  
Writing manifest to image destination
Storing signatures
49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c

Describe the results you expected:

Be able to use Podman in rootless mode for all images, mainly Arch Linux

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.1.0
API Version:  3.1.0
Go Version:   go1.16.2
Git Commit:   9f09fb62cba8f44c18eda84db3e72aab3ae76046-dirty
Built:        Wed Mar 31 11:46:48 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon pertence a conmon 1:2.0.27-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: 65fad4bfcb250df0435ea668017e643e7f462155'
  cpus: 4
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: arch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65537
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65537
  kernel: 5.11.11-arch1-1
  linkmode: dynamic
  memFree: 1937080320
  memTotal: 8222564352
  ociRuntime:
    name: runc
    package: /usr/bin/runc pertence a runc 1.0.0rc93-2
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc93
      commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
      spec: 1.0.2-dev
      go: go1.16.2
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns pertence a slirp4netns 1.1.9-1
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 12884897792
  swapTotal: 12884897792
  uptime: 52m 57.03s
registries:
  search:
  - docker.io
store:
  configFile: /home/rafael/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rafael/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/rafael/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.0
  Built: 1617202008
  BuiltTime: Wed Mar 31 11:46:48 2021
  GitCommit: 9f09fb62cba8f44c18eda84db3e72aab3ae76046-dirty
  GoVersion: go1.16.2
  OsArch: linux/amd64
  Version: 3.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

$ pacman -Qi podman
Name            : podman
Version         : 3.1.0-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  containers-common  device-mapper
                  iptables  libseccomp  runc  slirp4netns  libsystemd
                  fuse-overlayfs  libgpgme.so=11-64
Optional Deps   : podman-docker: for Docker-compatible CLI
                  btrfs-progs: support btrfs backend devices [installed]
                  catatonit: --init flag support
                  crun: support for unified cgroupsv2
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 72.86 MiB
Packager        : Morten Linderud <foxboron@archlinux.org>
Build Date      : Wed Mar 31 11:46:48 2021
Install Date    : Sun Apr 4 20:38:40 2021
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes (the commit tested was 72eb0005f) and yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Its a physical machine, running Arch Linux 64-bit, with containers-common installed.

Here is my rootless mode settings (See Podman in ArchWiki for the guide followed):

• kernel.unprivileged_userns_clone=1 already set by the stock Linux kernel in Arch Linux
• cgroups v2 already set since Systemd v248, and which is the version currently installed.
• subuid and subgid: both set with rafael:100000:65536 (my username and group)