podman: podman exec sometimes exits 137

I see this once in a while. Too infrequently to have a good reproducer. Only rootless so far:

$ while :;do bats --filter container /usr/share/podman/test/system/*sdnot*bats || break;done
...mostly works... then:
 ✗ sdnotify : container
   (from function `die' in file /usr/share/podman/test/system/helpers.bash, line 413,
    from function `run_podman' in file /usr/share/podman/test/system/helpers.bash, line 221,
    in test file /usr/share/podman/test/system/260-sdnotify.bats, line 163)
     `run_podman exec $cid touch /stop' failed with status 137
   $ podman rm --all --force
   $ podman ps --all --external --format {{.ID}} {{.Names}}
   $ podman images --all --format {{.Repository}}:{{.Tag}} {{.ID}}
   quay.io/libpod/testimage:20210610 9f9ec7f2fdef
   $ podman pull quay.io/libpod/fedora:31
   Trying to pull quay.io/libpod/fedora:31...
   Getting image source signatures
   Copying blob sha256:c28ace6b0c4ae099f6f81091731bdf41d9771d28bad96ae4a3507fe950560930
   Copying config sha256:a7a37f74ff864eec55891b64ad360d07020827486e30a92ea81d16459645b26a
   Writing manifest to image destination
   Storing signatures
   a7a37f74ff864eec55891b64ad360d07020827486e30a92ea81d16459645b26a
   $ podman run -d --sdnotify=container quay.io/libpod/fedora:31 sh -c printenv NOTIFY_SOCKET;echo READY;systemd-notify --ready;while ! test -f /stop;do sleep 0.1;done
   a98e2e8a856772144e2297d4dc3b2d21ffa7ab5ff6088a998168d08add9264f4
   $ podman logs a98e2e8a856772144e2297d4dc3b2d21ffa7ab5ff6088a998168d08add9264f4
   /tmp/podman_bats.HrMDlg/container.sock/notify
   READY
   $ podman logs a98e2e8a856772144e2297d4dc3b2d21ffa7ab5ff6088a998168d08add9264f4
   /tmp/podman_bats.HrMDlg/container.sock/notify
   READY
   $ podman exec a98e2e8a856772144e2297d4dc3b2d21ffa7ab5ff6088a998168d08add9264f4 touch /stop
   [ rc=137 (** EXPECTED 0 **) ]
   #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #| FAIL: exit code is 137; expected 0
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   # [teardown]
   $ podman pod rm --all --force
   $ podman rm --all --force
   a98e2e8a856772144e2297d4dc3b2d21ffa7ab5ff6088a998168d08add9264f4

1 test, 1 failure

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 58 (26 by maintainers)

Commits related to this issue

test/system/260-sdnotify.bats: fix test flake The `exec` session somestimes exits with 137 as the exec session races with the cleanup process of the exiting container. Fix the flake by running a det... — committed to vrothberg/libpod by vrothberg a year ago
system tests: safer container-stop signaling Having a container spin-wait on a /stop file, then exit, is unsafe: 'podman exec $ctr touch /stop' can get sucked into container cleanup before the exec t... — committed to edsantiago/libpod by edsantiago a year ago
system tests: safer container-stop signaling Having a container spin-wait on a /stop file, then exit, is unsafe: 'podman exec $ctr touch /stop' can get sucked into container cleanup before the exec t... — committed to edsantiago/libpod by edsantiago a year ago
system tests: safer container-stop signaling Having a container spin-wait on a /stop file, then exit, is unsafe: 'podman exec $ctr touch /stop' can get sucked into container cleanup before the exec t... — committed to edsantiago/libpod by edsantiago a year ago

Most upvoted comments

@sanjayts I’m going to bet that your issue is a too-tiny podman machine. My (limited) experience with Rust is that it demands a lot of RAM. Can you try bumping up the --memory in your VM?

edsantiago on Aug 29, 2022