podman: Podman systemd start (user) exit with status=125, but runs OK in command line

Issue Description

I’m trying to run a container as a systemd (user) service. Using the user account (photop) I’m able to start/stop the container with no issues. The container was created via:

podman run -d \
  --label "io.containers.autoupdate=registry" \
  --replace \
  --env-file $SCRIPT_DIR/env \
  --name photoprism \
  --security-opt seccomp=unconfined \
  --security-opt apparmor=unconfined \
  --annotation run.oci.keep_original_groups=1 \
  --userns=keep-id \
  -p 2342:2342 \
  -v /home/photop/storage:Z \
  -v /home/photop/data:/photoprism/originals:ro \
  docker.io/photoprism/photoprism

Steps to reproduce the issue

I’ve generated systemd via podman generate systemd --new photoprism > photop.service, which returned:

[Unit]
Description=Podman container-32078428ba9bbf3f623618f9764c146e757e583a060d63c09be0a318de9f53c5.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman run \
	--cidfile=%t/%n.ctr-id \
	--cgroups=no-conmon \
	--rm \
	--sdnotify=conmon \
	-d \
	--label io.containers.autoupdate=registry \
	--replace \
	--env-file /home/photop/env \
	--name photoprism \
	--security-opt seccomp=unconfined \
	--security-opt apparmor=unconfined \
	--annotation run.oci.keep_original_groups=1 \
	--userns=keep-id \
	-p 2342:2342 \
	-v /home/photop/storage:/photoprism/storage:Z \
	-v /home/photop/data:/photoprism/originals:ro docker.io/photoprism/photoprism
ExecStop=/usr/bin/podman stop \
	--ignore -t 10 \
	--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
	-f \
	--ignore -t 10 \
	--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target
photop@lubu3:~/.config/systemd/user$

1. Then I run systemctl --user start photoprism (after relaoding the systemd user daemon and ensuring the file is at ~/.config/systemd/user

2. Start up fails, logs are:

photop@lubu3:~/.config/systemd/user$ systemctl --user status photop.service
× p2.service - Podman container-32078428ba9bbf3f623618f9764c146e757e583a060d63c09be0a318de9f53c5.service
     Loaded: loaded (/home/photop/.config/systemd/user/p2.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 18:31:05 AEST; 12min ago
       Docs: man:podman-generate-systemd(1)
    Process: 1847485 ExecStart=/usr/bin/podman run --cidfile=/run/user/115/p2.service.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -d --label io.containers.autoupdate=registry --replace --env-file /home/photop/env --name photoprism --security-opt seccomp=unconfined --security-opt apparmor=unconfined --annotation run.oci.keep_original_groups=1 --userns=keep-id -p 2342:2342 -v /home/photop/storage:/photoprism/storage:Z -v /home/photop/data:/photoprism/originals:ro docker.io/photoprism/photoprism (code=exited, status=125)
    Process: 1847497 ExecStopPost=/usr/bin/podman rm -f --ignore -t 10 --cidfile=/run/user/115/p2.service.ctr-id (code=exited, status=0/SUCCESS)
   Main PID: 1847485 (code=exited, status=125)
        CPU: 105ms

Jun 28 18:31:05 lubu3 systemd[30735]: photop.service: Scheduled restart job, restart counter is at 5.
Jun 28 18:31:05 lubu3 systemd[30735]: Stopped Podman container-32078428ba9bbf3f623618f9764c146e757e583a060d63c09be0a318de9f53c5.service.
Jun 28 18:31:05 lubu3 systemd[30735]: photop.service: Start request repeated too quickly.
Jun 28 18:31:05 lubu3 systemd[30735]: photop.service: Failed with result 'exit-code'.
Jun 28 18:31:05 lubu3 systemd[30735]: Failed to start Podman container-3207842

However, if I copy paste the ExecStart line and run it manually (using the photop user account), it runs OK.

Describe the results you received

See above.

Describe the results you expected

Podman can run via systemd user service.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.7-0ubuntu22.04+obs15.48_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.56
    systemPercent: 0.13
    userPercent: 0.31
  cpus: 8
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: lubu3
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1004
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1004
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-73-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 301547520
  memTotal: 8263806976
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_101:1.8.4-0ubuntu22.04+obs55.14_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/1004/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1004/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.0.1-2_amd64
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 96989184
  swapTotal: 536866816
  uptime: 314h 20m 3.00s (Approximately 13.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/seb/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/seb/.local/share/containers/storage
  graphRootAllocated: 117234069504
  graphRootUsed: 73807609856
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1004/containers
  transientStore: false
  volumePath: /home/seb/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Ubuntu 22.04.2 LTS

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting