podman: podman system prune sometimes fails or podman stop doesn't wait for containers to stop

Issue Description

Sometimes when I run podman system prune -a -f immediately after stopping containers with podman stop, I get an error message and exit code 125.

If I run podman system prune -a -f again, I don’t get the error. So it seems like podman stop doesn’t wait for containers to stop.

Steps to reproduce the issue

1. Start a number of containers like this: for i in {1..10} ; do podman run -id --rm registry.opensuse.org/opensuse/tumbleweed ; done
2. Stop them all and immediately prune them like this: podman ps -q | xargs -r podman stop ; echo STOPPED ; podman system prune -a -f ; echo $?

Repeat the above steps until you get an error.

Describe the results you received

localhost:~ # for i in {1..10} ; do podman run -id --rm registry.opensuse.org/opensuse/tumbleweed ; done
Trying to pull registry.opensuse.org/opensuse/tumbleweed:latest...
Getting image source signatures
Copying blob 7ab641c6960d done  
Copying config 5cb33712cf done  
Writing manifest to image destination
Storing signatures
58174dcb7a6cb7959ab71d2404e4fa964fd508e0523511bb9bf7abc58dc834cb
b55c3ea414d8812eff1652108b386eaf7e5b0e54fbbbf35baf01bb1a30f64c09
8a929fbb5716f34eb4c7407daeb9123ca67aacd65a257558b8e70983b68de752
acaeef00e4b7f6b0c6fec052b0a1195324374502263715455118fcfb2bc77498
7eadd0bff219e7665c78b508d952f9a6331cfdf7c93bec7954bc8ff1d1f58b1a
7c4a93bc320e021042b55566b6f237c70465c281cab1f4c5aca255770626cf36
6a4ea4991361231aa7609456ac53e56a1bf537be3c6b9920c179f5f46470de1c
cfabdc30e28e0b0837521fdee22085be8e5c352f9a6e31938d3c025be719db51
b5c6c954c469b4d946dfb7d5f387328d20cfdeb7950e54e6311b92f1a073e651
486b67725af81c656ec62250b1a101eb5ad299d81721d25840af28d40cdd274d

# podman ps -q | xargs -r podman stop ; echo STOPPED ; podman system prune -a -f
WARN[0010] StopSignal SIGTERM failed to stop container kind_bose in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container eloquent_taussig in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container sweet_elgamal in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container charming_galois in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container optimistic_villani in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container intelligent_pasteur in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container boring_cori in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container nice_spence in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container hardcore_wright in 10 seconds, resorting to SIGKILL 
WARN[0010] StopSignal SIGTERM failed to stop container upbeat_payne in 10 seconds, resorting to SIGKILL 
7c4a93bc320e
6a4ea4991361
486b67725af8
7eadd0bff219
b5c6c954c469
8a929fbb5716
acaeef00e4b7
cfabdc30e28e
58174dcb7a6c
b55c3ea414d8
STOPPED
Deleted Containers
b5c6c954c469b4d946dfb7d5f387328d20cfdeb7950e54e6311b92f1a073e651
Error: container b5c6c954c469b4d946dfb7d5f387328d20cfdeb7950e54e6311b92f1a073e651 does not exist in database: no such container

# echo $?
125

# podman system prune -a -f
Total reclaimed space: 0B

Describe the results you expected

I expect that podman stop waits for containers to really stop so podman system prune doesn’t exit with 125.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.4.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
    idlePercent: 99.88
    systemPercent: 0.06
    userPercent: 0.06
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: '"opensuse-microos"'
    version: "20230626"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.3.9-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 32693002240
  memTotal: 33643749376
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.7-1.3.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.7
      commit: v1.1.7-0-g860f061b76bb
      spec: 1.0.2-dev
      go: go1.20.5
      libseccomp: 2.5.4
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-1.3.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 3h 3m 58.00s (Approximately 0.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 5378125824
  graphRootUsed: 369967104
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685318400
  BuiltTime: Mon May 29 02:00:00 2023
  GitCommit: ""
  GoVersion: go1.20.5
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

QEMU VM with 32GB RAM running openSUSE MicroOS.

Additional information

This bug was found thanks to openQA automation: https://openqa.suse.de/tests/11383574/logfile?filename=serial_terminal.txt