finit: regression? 4.9, 4.10rc1 crash on reboot
situation: we have a fairly straightforward finit configuration set which runs up a few services,
sequenced using a few custom conditions, and one sys/pwr/fail task that calls initctl poweroff.
with 4.8, both power-fail-triggered and explicitly triggered reboots and shutdowns work fine and reliably.
with 4.9 and the same configuration rebooting doesn’t work.
using 4.9, initctl debug followed by initctl reboot:
the last lines displayed are
finit[1]: sm_step():state: runlevel/wait, runlevel: 6, newlevel: -1, teardown: 1, reload: 0
finit[1]: sm_step():All services have been stopped, calling runlevel change hooks ...
...a list of many things stopped or halted...
finit[1]: plugin_unregister():bootmisc exiting ...
immediately followed by a panic: ‘finit[1]:[ 177.502157] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b’
using 4.9, initctl debug, then triggering the sys/pwr/fail with SIGPWR: and the unit shuts down successfully;
the log output is very similar but following the bootmisc exiting, we get alsa-utils, tty, procps … exiting lines and the shutdown progresses as expected.
active plugins: alsa-utils.so netlink.so sys.so bootmisc.so pidfile.so tty.so procps.so urandom.so hook-scripts.so rtc.so usr.so hook scripts: one in sys/down (but things don’t progress that far in the failure scenario) no post: scripts whatsoever.
i’ve tried to diff the debug output of a succeeding vs. failing shutdown run, and apart from the expected numeric pid and arg 0xwhatever differences, there doesn’t seem to be anything majorly different. after finit logs ‘All services have been stopped’, the listing of service states looks pretty much the same (except for the powerfail condition being on vs. off). i’m attaching both those logs.
joachim, any ideas as to what the problem might be? any suggestions for debugging this further would be highly appreciated.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (24 by maintainers)
@troglobit - all good here, no crash on reboot
as always thanks for your efforts 🙇♂
i’ve run a few more checks and with https://github.com/troglobit/finit/commit/a23e152f6b2c883130bb0a4bfa55ef304294a369 shutdown works fine now. thank you!
On Fri, 28 Feb 2025 02:23:46 -0800, Joachim Wiberg writes:
i’ll check this first thing monday and report back.
– Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + https://snafu.priv.at/ To rephrase, spam is not the answer. Spam is the question. Death is not the answer, but pretty close to it. – Vadik
@aanderse: to sidestep this issue i’ve patched out the plugin_exit altogether, and that works fine for our use case (which is physical machines, not containers); see attached.
no-plugin-unload.txt
once joachim is well again he may very well come up with a good reason why finit does need to run plugin_exit, in which case the plugin queue iteration needs to be looked at further.
indeed, puzzling why that raise of the condition makes any difference at all. (i ran the TERM vs. INT-then-TERM experiment a good 10 times, just to be sure that it’s not a one-off fluke…)
my 4.8 and the 4.9 and 4.10 test builds are all against the same baseline chroot, with the usual debianish reproducible build settings so i don’t think a rebuild of 4.8 will make it fail - but i’ll try that tomorrow.
newer libs: i’ve had zero success building against debian’s libuev3 (=2.4.0), which is why my baseline chroot is still using bullseye. any finit that i’ve build against that newer libuev3 fails to progress beyond starting of an mdev. i may retry with local builds of the newest libuev and libite, but that’ll have to wait a few days.
i’ll certainly check with gdb, but again this will take a few days.
let’s leave the bug open for now, but given the odd nature of it and the fact that 4.8 works fine i don’t think it’s high priority.
On Sun, 23 Feb 2025 21:44:50 -0800, Joachim Wiberg writes:
possibly relevant in that context: both 4.8 and 4.9 were built locally, and both against the same combo of libite5 2.2.0, libuev2 2.3.1 and libc6 2.31 (ie. debian bullseye, not bookworm).
thanks, i’ll try that tomorrow and will report back soonest.
– Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + https://snafu.priv.at/ All that blue light from Orthanc at night? That was Saruman, trying to moderate – news.admin.palantir-abuse.sightings.