salt: minion not returning, apt-get defunct

I am using SaltStack 0.17.4 on Debian 7.3 for master and minion with great success, but have run into a problem.

Running “salt-call -l debug state.highstate” on the minion is successful, but when running “salt ‘host’ state.highstate” from the master, or using the daemon with “startup_states: highstate” on the minion causes apt-get to become defunct, and I suspect one of the package scripts might be causing this.

I am trying to install Proxmox VE as per http://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Wheezy and have traced the problem to the following section in my state file:

pve:
  pkg:
    - installed
    - pkgs:
      - proxmox-ve-2.6.32
      - lvm2
      - postfix
      - ksm-control-daemon
      - vzprocps
      - open-iscsi
      - bootlogd

salt --versions-report:
Salt: 0.17.4
Python: 2.7.3 (default, Jan  2 2013, 13:56:14)
Jinja2: 2.6
M2Crypto: 0.21.1
msgpack-python: 0.1.10
msgpack-pure: Not Installed
pycrypto: 2.6
PyYAML: 3.10
PyZMQ: 13.1.0
ZMQ: 3.2.3

Any thoughts?

About this issue

Original URL
State: closed
Created 10 years ago
Comments: 32 (12 by maintainers)

Most upvoted comments

@tsia for mysql 5.7 we were able to fix that by changing init script a bit, it’s around line 162 on ubuntu

             su - mysql -s /bin/bash -c "(mysqld_safe 2>&1)> /dev/null &"

michalsuba on Jan 28, 2016

Personally I would be fine even with having an additional parameter, like force_detach or legacy or really_bad_service - as long as I can point people to documentation and say, “hey, look here, try that parameter”.

It sounds wrong to me adding another workaround and letting people investigate which mal-behaving startup script exhibits this behavior and needs a workaround - instead of fixing the problem where the problem lies, i.e. in the salt minion, not properly catching SIGCHLD signal and closing all pipes passed to the exited subprocess.

I’ll repeat what I wrote in https://github.com/saltstack/salt/issues/33442#issuecomment-261711016

To make myself explicit: services which behave as described above (a process terminates but passes its descriptors to other processes before exiting) may indeed not be behaving as nice citizens, but the bug lies entirely in SaltStack.

When SaltStack spawns some process (i.e. runs a command) and creates a pipe to its stdout and stderr, SaltStack must establish a SIGCHLD handler. When the spawned process terminates, any I/O on such pipes must be terminated (even if the pipe is still open on the other side). Promptly reclaiming (waitpid) the exit status of a terminated child process also avoids having zombies lying around.

MarkMartinec on Feb 24, 2017