salt: Background process dies with salt-minion

Description of Issue/Question

I have a start.sh script which looks like

    setsid screen -AmdS "myapp" bash start2.sh &> /dev/null

The newly spawned process shows no relation to salt-minion (process group, parent PID etc.)

The problem is that the background process gets a SIGTERM when salt-minion is restarted and dies.

This is seriously annoying. There are no zombie processes created by salt.

  PID  PPID  PGID   SID COMMAND
26911     1 26911 26911 screen
26912 26911 26912 26912 java
(java is the app process)

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Steps to Reproduce Issue

The start.sh script is invoked via cmd.run (it’s important for it to be run via master to trigger the issue, no problems when using salt-call)

(Include debug logs if possible and relevant.)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2016.11.1-8-g48f1629
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.5
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.3.9
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 3.6.0 (default, Jan 16 2017, 12:12:55)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.0
 
System Versions:
           dist: arch Arch Linux
        machine: x86_64
        release: 3.14.32-xxxx-grs-ipv6-64
         system: Linux
        version: arch Arch Linux

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 27 (27 by maintainers)

Most upvoted comments

It is killed off by systemd.

If you check systemd-cgls, you can see the screen has been reparented from the minion process, but is still under the salt-minion.service.

└─system.slice
  ├─salt-minion.service
  │ ├─6058 /usr/bin/python /usr/bin/salt-minion
  │ ├─6068 /usr/bin/python /usr/bin/salt-minion
  │ ├─6072 /usr/bin/python /usr/bin/salt-minion
  │ ├─6274 SCREEN -AdmS name bash
  │ └─6275 bash

We need to have a cmdmod command that runs the command through systemd-run like your package managers modules do. https://github.com/saltstack/salt/blob/develop/salt/modules/yumpkg.py#L1438

gtmanfred on Feb 9, 2017