Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

As of recent linuxserver containers have stopped working with the --user flag. I believe the proper capabilities are not being added with the cap-add flag. Testing with capeff shows that no capabilities are being added. This wasn’t always the case, I used to be able to run pods with the ‘–user’ flag with no issues. This is happening with at least the qbittorrent, radarr, sonarr, bazarr and jellyfin images. All images was working fine a few months ago.

I am not really sure how to properly diagnose the issue. However, I was able to find the first image released that stopped working with the --user tag for the qbittorrent image. It will be image 4.4.3.1-r2-ls202 which make image 4.4.3.1-r1-ls201 to be the last working image.

I have no issue running the latest container images without the --user flag. Running --cap-drop and --cap-add actually do work when not using the --user flag.

Steps to reproduce the issue:

1. Pull recent linuxserver container image, for example qbittorrent image

2. Run podman with --user flag adding necessary capabilities with cap-add

podman run -d \
  --name=qbittorrent \
  --user 1000:1000 \
  -p 8080:8080 \
  --cap-add=CHOWN,DAC_OVERRIDE,FOWNER,KILL,SETGID,SETUID \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=America/New_York \
  -e WEBUI_PORT=8080 \
  --restart unless-stopped \
  linuxserver/qbittorrent:latest

Describe the results you received: Unable to access webui. Container does not work.

Output of podman logs -f qbittorrent

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service 00-legacy: starting
s6-rc: info: service 00-legacy successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/01-envfile
cont-init: info: /etc/cont-init.d/01-envfile exited 0
cont-init: info: running /etc/cont-init.d/01-migrations
[migrations] started
[migrations] no migrations found
cont-init: info: /etc/cont-init.d/01-migrations exited 0
cont-init: info: running /etc/cont-init.d/02-tamper-check
cont-init: info: /etc/cont-init.d/02-tamper-check exited 0
cont-init: info: running /etc/cont-init.d/10-adduser
groupmod: /etc/group.69: Permission denied
groupmod: cannot lock /etc/group; try again later.
usermod: /etc/passwd.70: Permission denied
usermod: cannot lock /etc/passwd; try again later.

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    1001
-------------------------------------

chown: changing ownership of '/app': Operation not permitted
chown: changing ownership of '/config': Operation not permitted
chown: changing ownership of '/defaults': Operation not permitted
cont-init: info: /etc/cont-init.d/10-adduser exited 1
cont-init: info: running /etc/cont-init.d/30-config
/package/admin/s6-overlay-3.1.0.1/etc/s6-rc/scripts/cont-init: line 14: /etc/cont-init.d/30-config: Permission denied
cont-init: info: /etc/cont-init.d/30-config exited 126
cont-init: info: running /etc/cont-init.d/90-custom-folders
mkdir: cannot create directory ‘/config/custom-cont-init.d’: Permission denied
mkdir: cannot create directory ‘/config/custom-services.d’: Permission denied
chown: cannot access '/config/custom-cont-init.d': No such file or directory
chown: cannot access '/config/custom-services.d': No such file or directory
cont-init: info: /etc/cont-init.d/90-custom-folders exited 1
cont-init: info: running /etc/cont-init.d/99-custom-files
[custom-init] no custom files found exiting...
cont-init: info: /etc/cont-init.d/99-custom-files exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service init-mods: starting
s6-rc: info: service init-mods successfully started
s6-rc: info: service init-mods-package-install: starting
s6-rc: info: service init-mods-package-install successfully started
s6-rc: info: service init-mods-end: starting
s6-rc: info: service init-mods-end successfully started
s6-rc: info: service init-services: starting
s6-rc: info: service init-services successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun qbittorrent (no readiness notification)
s6-supervise qbittorrent (child): fatal: unable to exec run: Permission denied
s6-supervise qbittorrent: warning: unable to spawn ./run - waiting 10 seconds
s6-rc: info: service legacy-services successfully started
s6-rc: info: service 99-ci-service-check: starting
[ls.io-init] done.
s6-rc: info: service 99-ci-service-check successfully started
s6-supervise qbittorrent (child): fatal: unable to exec run: Permission denied
s6-supervise qbittorrent: warning: unable to spawn ./run - waiting 10 seconds

output of podman top -l capeff

EFFECTIVE CAPS
none
none
none
none
none
none
none

Describe the results you expected:

For the container to work properly, access webui. Example running container with most recent working image linuxserver/qbittorrent:4.4.3.1-r1-ls201

output of podman logs -f qbittorrent

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing... 
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 01-migrations: executing... 
[migrations] started
[migrations] no migrations found
[cont-init.d] 01-migrations: exited 0.
[cont-init.d] 02-tamper-check: executing... 
[cont-init.d] 02-tamper-check: exited 0.
[cont-init.d] 10-adduser: executing... 

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/


Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    1000
User gid:    1000
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 30-config: executing... 
[cont-init.d] 30-config: exited 0.
[cont-init.d] 90-custom-folders: executing... 
[cont-init.d] 90-custom-folders: exited 0.
[cont-init.d] 99-custom-files: executing... 
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-files: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.

******** Information ********
To control qBittorrent, access the WebUI at: http://localhost:8080

output of podman top -l capeff

EFFECTIVE CAPS
CHOWN,DAC_OVERRIDE,FOWNER,KILL,SETGID,SETUID
CHOWN,DAC_OVERRIDE,FOWNER,KILL,SETGID,SETUID
CHOWN,DAC_OVERRIDE,FOWNER,KILL,SETGID,SETUID
CHOWN,DAC_OVERRIDE,FOWNER,KILL,SETGID,SETUID

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 4.1.1

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.2-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.2, commit: unknown'
  cpuUtilization:
    idlePercent: 63.57
    systemPercent: 9.53
    userPercent: 26.9
  cpus: 12
  distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20220812"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.19.0-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 175075328
  memTotal: 4086444032
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.3-2.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.3
      commit: v1.1.3-0-ga916309fff0f
      spec: 1.0.2-dev
      go: go1.18.3
      libseccomp: 2.5.4
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.11-1.6.x86_64
    version: |-
      slirp4netns version 1.1.11
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.4
  swapFree: 871059456
  swapTotal: 1339006976
  uptime: 31m 58.73s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/test/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/test/.local/share/containers/storage
  graphRootAllocated: 19593691136
  graphRootUsed: 6826401792
  graphStatus:
    Build Version: Btrfs v5.18.1
    Library Version: "102"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 15
  runRoot: /run/user/1000/containers
  volumePath: /home/test/.local/share/containers/storage/volumes
version:
  APIVersion: 4.1.1
  Built: 1656633600
  BuiltTime: Thu Jun 30 20:00:00 2022
  GitCommit: ""
  GoVersion: go1.16.15
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.1.1-2.1.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Checked troubleshooting. Have tried most recent github version.

Additional environment details (AWS, VirtualBox, physical, etc.):