podman: Active podman process blocks system reboot/shutdown

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

An active podman process is unable to be cleanly stopped by systemd reboot/shutdown, and thus has to be killed after the 2min grace period expires.

Steps to reproduce the issue:

podman run -it docker.io/library/busybox
Inside the container: sleep infinity
Reboot the system

Describe the results you received:

Shutdown procedure hangs for ~2 minutes because podman can’t be stopped. Then podman is killed and shutdown is complete.

Describe the results you expected:

The podman container to be cleanly terminated as the system shuts down.

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.1.0-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Experienced this issue on Fedora Workstation 36 and Fedora Silverblue 36.

Downstream bug reports:

podman: https://bugzilla.redhat.com/show_bug.cgi?id=2084498
toolbox: https://bugzilla.redhat.com/show_bug.cgi?id=2081664

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 34 (22 by maintainers)

Commits related to this issue

health check: ignore dependencies of transient systemd units/timers When stopping the transient systemd timer/unit which powers running health checks, make sure to ignore its dependencies. It turns ... — committed to vrothberg/libpod by vrothberg 2 years ago

Most upvoted comments

https://github.com/containers/podman/pull/16785 fixes the issue and will make it into Podman 4.4.

vrothberg on Dec 9, 2022

I wished having found more time to work on this bug. One thing I noticed while debugging is that we’re stuck on stopping the transient health-check timer.

I hope to find some time tomorrow.

vrothberg on Dec 8, 2022

@vrothberg, you can test it with this container service on Silverblue. It takes 2 min to reboot/shutdown.

syncthing-test.service
- systemctl --user start syncthing-test.service
- Then reboot.the system.

PS: This is syncthing official container, I didn’t add any volume or any published port.

Dockerfile: https://github.com/syncthing/syncthing/blob/main/Dockerfile

Only way to reboot this systemd container service without waiting is use --no-healthcheck on podman args.

# autogenerated by Podman 4.3.1
# Tue Dec  6 16:27:12 +03 2022

[Unit]
Description=Podman syncthing-test.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=no
TimeoutStopSec=70
ExecStartPre=/bin/rm \
    -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run \
    --cidfile=%t/%n.ctr-id \
    --cgroups=no-conmon \
    --rm \
    --sdnotify=conmon \
    --replace \
    --detach \
    --name syncthing-test docker.io/syncthing/syncthing
ExecStop=/usr/bin/podman stop \
    --ignore -t 10 \
    --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
    -f \
    --ignore -t 10 \
    --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

queeup on Dec 6, 2022