podman: Podman rm (--force) does not work anymore ("container state improper"/"invalid argument" when unmounting) and start neither

Issue Description

Describe the bug Somehow I get the error:

Error: container “nextcloud_[…]” is mounted and cannot be removed without using force: container state improper

…when trying to remove containers.

Steps to reproduce the issue

$ podman-compose -p nextcloud down
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.1
** excluding:  set()
podman stop -t 10 nextcloud_caddy_1
Error: no container with name or ID "nextcloud_caddy_1" found: no such container
exit code: 125
podman stop -t 10 nextcloud_cron_1
Error: no container with name or ID "nextcloud_cron_1" found: no such container
exit code: 125
podman stop -t 10 nextcloud_nc_1
Error: no container with name or ID "nextcloud_nc_1" found: no such container
exit code: 125
podman stop -t 10 nextcloud_db_1
Error: no container with name or ID "nextcloud_db_1" found: no such container
exit code: 125
podman stop -t 10 nextcloud_redis_1
Error: no container with name or ID "nextcloud_redis_1" found: no such container
exit code: 125
podman rm nextcloud_caddy_1
Error: no container with ID or name "nextcloud_caddy_1" found: no such container
exit code: 1
podman rm nextcloud_cron_1
Error: no container with ID or name "nextcloud_cron_1" found: no such container
exit code: 1
podman rm nextcloud_nc_1
Error: container "nextcloud_nc_1" is mounted and cannot be removed without using force: container state improper
exit code: 2
podman rm nextcloud_db_1
Error: container "nextcloud_db_1" is mounted and cannot be removed without using force: container state improper
exit code: 2
podman rm nextcloud_redis_1
Error: container "nextcloud_redis_1" is mounted and cannot be removed without using force: container state improper
exit code: 2
$ podman rm nextcloud_db_1
Error: container "nextcloud_db_1" is mounted and cannot be removed without using force: container state improper

When trying to start the container I get this error then:

$ podman-compose --in-pod=0 -p nextcloud up -d
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.1
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=nextcloud', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
podman pod create --name=pod_nextcloud --infra=false --share=
Error: adding pod to state: name "pod_nextcloud" is in use: pod already exists
exit code: 125
[…]
Error: creating container storage: the container name "nextcloud_redis_1" is already in use by 4dbd88724af1ee89d859c6b2dfebb89f95cf6358503e09a8763009877a4830cb. You have to remove that container to be able to reuse that name: that name is already in use
exit code: 125
podman start nextcloud_redis_1
Error: no container with name or ID "nextcloud_redis_1" found: no such container
exit code: 125
[…]
Error: creating container storage: the container name "nextcloud_db_1" is already in use by 2760fa4a652ba952ef5270d256c658dd3f4455d96fe7554abdb13bbfbdbd6c19. You have to remove that container to be able to reuse that name: that name is already in use
exit code: 125
podman start nextcloud_db_1
Error: no container with name or ID "nextcloud_db_1" found: no such container
exit code: 125
[…]

Then there are dependency errors depending on the mentioned containers to start.

The thing is I see nothing of that running?

$ podman ps -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$ podman pod ls
POD ID        NAME           STATUS      CREATED      INFRA ID    # OF CONTAINERS
562b052bdab9  pod_nextcloud  Created     4 hours ago              0

Also the container that is said to use the name, is not there?

$ podman inspect 4dbd88724af1ee89d859c6b2dfebb89f95cf6358503e09a8763009877a4830cb
[]
Error: no such object: "4dbd88724af1ee89d859c6b2dfebb89f95cf6358503e09a8763009877a4830cb"

I can remove the pod, but it does not help:

$ podman pod rm pod_nextcloud 
562b052bdab9c31692403405579935979a7026f1945bbc0eb0f1594a2b80b546
$ podman inspect nextcloud_db_1
[]
Error: no such object: "nextcloud_db_1"

Describe the results you received

I cannot stop or start the containers.

Describe the results you expected

I should somehow be able to force/fix that. I have no idea what is “improper” nor how to fix it.

podman info output

$ podman-compose version
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.1
podman-compose version 1.0.6
podman --version 
podman version 4.6.1
exit code: 0
$ podman version
Client:       Podman Engine
Version:      4.6.1
API Version:  4.6.1
Go Version:   go1.20.7
Built:        Fri Aug 11 00:07:53 2023
OS/Arch:      linux/amd64
$ podman info
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 98.31
    systemPercent: 0.82
    userPercent: 0.86
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  freeLocks: 615
  hostname: minipure
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 6.4.7-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 60165713920
  memTotal: 67283185664
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.6-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/user/1002/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    path: /run/user/1002/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 4h 27m 21.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/****/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 4
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/****/.local/share/containers/storage
  graphRootAllocated: 999650168832
  graphRootUsed: 46548176896
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 12
  runRoot: /run/user/1002/containers
  transientStore: false
  volumePath: /var/home/****/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1691705273
  BuiltTime: Fri Aug 11 00:07:53 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1
$ rpm -q podman
podman-4.6.1-1.fc38.x86_64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes kinda

Additional environment details

• OS: Linux Fedora CoreOS v38.20230819.3.0
• podman version: 4.6.1
• podman compose version: (git hex) 1.0.6

Additional information

Docker-compose v’3.7’ – exact same YAML started before without any problems.

I tested the echo example here and it did work, I have no idea what’s wrong.

Cross-posted as https://github.com/containers/podman-compose/issues/767 and https://github.com/coreos/fedora-coreos-tracker/issues/1572