podman: `OCI permission denied` if volume is located within user mount

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I’m trying to put Podman bind volumes of a rootless container within a gocryptfs user mount run by the same user. Everything works fine if the volume bind is located outside the mount but it fails with OCI permission denied if it is within the mount.

Steps to reproduce the issue:

create a gocryptfs mount
try to mount a volume within the mount

But I created a repository with Vagrant an Ansible to reproduce the issue reliably since this is probably better to debug.

Describe the results you received:

The container isn’t starting and complains with

Error: unable to start container "…": error stat'ing file `/home/podman_user/decrypted/postgres`: Permission denied: OCI permission denied

which seem to be a common error message.

Describe the results you expected:

The container runs.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

Click to show

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 1
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: ubuntu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-27-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 28434432
  memTotal: 490627072
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 0
  swapTotal: 0
  uptime: 13m 34.14s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/podman_user/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman_user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1001/containers
  volumePath: /home/podman_user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

Package info (output of apt list podman):

podman/jammy,now 3.4.4+ds1-1ubuntu1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No, I don’t know how to properly get a more recent version on Ubuntu 22.04 since the PPA isn’t available for this version any more. Is there any new, production-ready and reliable variant to do this?

Additional environment details (AWS, VirtualBox, physical, etc.): Libvirt

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 25 (11 by maintainers)

Most upvoted comments

another thing to check: the FUSE file system must be mounted with allow_other otherwise other users cannot access it.

giuseppe on Jun 8, 2022

The error isn’t specific to FUSE file systems, so that’s not always the fix. I don’t this because I was experiencing the issue on am EXT4 filesystem that doesn’t even have that mount option. It’s also the same error if you don’t have permissions to the mount, or to the directory you’re trying to mount.

mtalexan on Feb 3, 2023

hmmm getting similar issue. I’m using podman on macos, not sure what I do allow_other or where to execute. It would be great if someone can give more hint. Thanks

@kkharji
When you’re calling to mount the FUSE filesystem, you need to make sure to add the extra mount option allow_other to the mount command. How you do that depends on the particulars of your system type and how you mounted the FUSE filesystem.

mtalexan on Feb 3, 2023

I’m also still getting permission issues with :U volume mounts or when trying to do podman unshare.

But I’ll open a separate issue for this at one point since it’s slightly different.

alexanderadam on Jan 29, 2023