podman: podman with custom `static_dir` slow to tear-down containers

Issue Description

Whilst podman is able to start containers from even very large images fast enough that it feels interactively instant, shutting them down again takes what feels like a long time.

I wonder whether this is because I’m running the btrfs storage-driver, because I’m not running on a system managed by systemd (and so perhaps some call-back is failing?), or whether some default timeout is being adhered to even if the container processes have exited near the start of the timeout period?

Steps to reproduce the issue

$ sudo podman run -it --rm docker.io/alpine:latest ; date
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 7264a8db6415 skipped: already exists
Copying config 7e01a0d0a1 done
Writing manifest to image destination
/ # date ; exit
Tue Sep 12 08:40:27 UTC 2023
Tue 12 Sep 09:40:47 BST 2023

Describe the results you received

There is a 20s pause between the container process exiting and the container itself having been fully torn-down.

N.B. This is not a new problem - this system has been upgraded from podman 2.x with the VFS storage-driver, but has always been slow to tear-down containers.

Describe the results you expected

In other scenarios, if the process within the container exits then the container is torn-down effectively instantly.

podman run has a --stop-timeout which apparently defaults to 10 seconds, but the stop delay is actually twice that - and in this case the container process has itself exited, so I’d imagine that any timeout dependent on processes within the container (as opposed to related host processes such as crun/runc) should not be adhered to?

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.1.8
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 95.44
    systemPercent: 2.05
    userPercent: 2.5
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: gentoo
    version: "2.14"
  eventLogger: file
  freeLocks: 2021
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.2-gentoo
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 33188667392
  memTotal: 67331043328
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: app-containers/aardvark-dns-1.7.0
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: app-containers/netavark-1.7.0
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: app-containers/crun-1.9
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /var/run/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 42949652480
  swapTotal: 42949652480
  uptime: 8h 6m 42.00s (Approximately 0.33 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
  - registry.fedoraproject.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 25
    paused: 0
    running: 24
    stopped: 1
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /mnt/podman-storage
  graphRootAllocated: 2000397795328
  graphRootUsed: 73760190464
  graphStatus:
    Build Version: Btrfs v6.3.3
    Library Version: "102"
  imageCopyTmpDir: /var/tmp/.private/root
  imageStore:
    number: 120
  runRoot: /var/run/podman
  transientStore: false
  volumePath: /space/podman/volumes
version:
  APIVersion: 4.6.2
  Built: 1693253621
  BuiltTime: Mon Aug 28 21:13:41 2023
  GitCommit: 5db42e86862ef42c59304c38aa583732fd80f178
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Host is running openrc, not systemd.

Additional information

The size of the image (within reason) does not seem to have a significant effect on the tear-down time - for a 1.08GB image, the result is:

$ sudo podman run -it --rm --entrypoint /bin/sh localhost/gentoo-build:latest ; date
/ # date ; exit
Tue Sep 12 10:03:07 BST 2023
Tue 12 Sep 10:03:27 BST 2023

… so the delay appears to be a consistent 20s.