podman: podman 3.4.4 and Fedora 40 filesystem permission problems

Issue Description

I’ve been using podman to run the test suite for Lorax and weldr-client for a number of years with no problems until recently. Everything works fine locally using Fedora 38 and podman 4.7.2, the problem is that I use the same setup in my GitHub actions and with Fedora Rawhide (f40) I’ve started getting permission denied errors when rsync and tar try to modify files in the container.

Steps to reproduce the issue

I’ve setup a simplified repo here that demonstrates what I’m seeing: https://github.com/bcl/podman-fedora-action/actions it uses rsync to make a copy of the source tree in the container. The failed action run is with rawhide, the passing one is with Fedora 39.

Describe the results you received

You can see it complain about permissions in the output:

podman run --rm -v `pwd`/.test-results/:/test-results \
	-v `pwd`:/lorax-ro:ro --security-opt label=disable \
	--env RUN_TESTS="ci" \
	welder/lorax-tests:rawhide make test-in-copy
rsync -a --exclude=.git /lorax-ro/ /lorax/
rsync: [generator] failed to set permissions on "/lorax/.github": Operation not permitted (1)
...

If I add --no-perms to rsync it runs just fine.

Describe the results you expected

I expected no permission errors 😃

podman info output

Run podman info && make test-in-podman
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 4
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: fv-az777-825
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 127
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.2.0-1018-azure
  linkmode: dynamic
  logDriver: journald
  memFree: 14859001856
  memTotal: 16757788672
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 2m 8.92s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/runner/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  volumePath: /home/runner/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

This is running via GitHub actions on ubuntu-latest (22.04) with podman 3.4.4, I think it’s in a container but I really don’t know how GitHub runs the action.

Additional information

If I switch the container to use Fedora 39 (as you can see in the passing actions in my example repo) it works fine. So I suspect this is some kind of interaction between podman 3.4.4 on Ubuntu and Fedora Rawhide (40) but I don’t know how to track it any further than that.

With rsync I can work around it by passing --no-perms but after seeing similar issues with tar in the weldr-client project I figured it might be better to track down what’s actually happening.