core: pkg stalls on exit

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[x] I have searched the existing issues and I’m convinced that mine is new.

Describe the bug Although core might not be the best place to open this report, I thought to add it here so we don’t forget to look at this (preferably before a next release).

Since pkg 1.15.6 a package install sometimes freezes on exit. At first I thought it was related to syslog-ng spawning it’s own workers which pkg didn’t know about, but when disabling the lockout handler the issue seems to remain.

To Reproduce Steps to reproduce the behavior:

Easiest option to reproduce is to go to a small plugin and make upgrade a couple of times, after a couple of times it always seems to freeze. Currently I don’t suspect syslog-ng of not wanting to restart, because it hangs after configuring system logging.

Configuring system logging...done.

When running the command in truss (truss make upgrade), it doesn’t seem to be stalling on my end, which makes debugging a bit difficult. When looking at the hanging process with truss, it shows the following:

# fstat -p 42816 -v
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     pkg        42816 text /        7065390 -rwxr-xr-x  3069048  r
root     pkg        42816 ctty /dev        132 crw--w----   pts/1 rw
root     pkg        42816   wd /        7713449 drwxr-xr-x     512  r
root     pkg        42816 root /             2 drwxr-xr-x    1024  r
root     pkg        42816    0 /dev        132 crw--w----   pts/1 rw
root     pkg        42816    1 /dev        132 crw--w----   pts/1 rw
root     pkg        42816    2 /dev        132 crw--w----   pts/1 rw
root     pkg        42816    3 /             2 drwxr-xr-x    1024  r
root     pkg        42816    4 /        2969888 drwxr-xr-x     512  r
root     pkg        42816    5 /        2969889 -rw-r--r--  27588608 rw
root     pkg        42816    6 /        7729592 -rw-r--r--    2888  r
root     pkg        42816    7 /             2 drwxr-xr-x    1024  r
root     pkg        42816    8* local seqpak fffff800043b2500 <-> fffff800a7578500

# truss -p 42816
read(8,"\0\0\0",8192)				 = 3 (0x3)
SIGNAL 18 (SIGTSTP) code=SI_KERNEL
read(8,0x3397b09e000,8192)			 = 0 (0x0)
close(8)					 = 0 (0x0)
access("/etc/localtime",R_OK)			 = 0 (0x0)

Expected behavior It (pkg) should probably just exit, but if restarting services in pkg causes issues more frequently, we might look into alternatives (detach the restart process?). I’m not 100% sure how this is supposed to work, for some reason I haven’t seen it crash when restarting configd for example.

Screenshots

Relevant log files

Additional context

Environment

OPNsense 20.7.3 (amd64, OpenSSL).