salt: Salt minion on FreeBSD some services hang on restart

Description of Issue

The salt minion frequently hangs when restarting some services. Some more than others. One that I can regularly reproduce from the minion (but not when I run from the command line) is icinga2:

[INFO    ] Loading fresh modules for state activity
[INFO    ] Completed state [icinga2] at time 20:03:14.168931 (duration_in_ms=27004.632)
[INFO    ] Running state [icinga2] at time 20:03:14.172583
[INFO    ] Executing state service.running for [icinga2]
[INFO    ] Executing command '/usr/sbin/service -l' in directory '/root'
[INFO    ] Executing command '/usr/sbin/service icinga2 onestatus' in directory '/root'
[INFO    ] Executing command '/usr/sbin/service -l' in directory '/root'
[INFO    ] Executing command '/usr/sbin/service icinga2 rcvar' in directory '/root'
[INFO    ] Executing command '/usr/sbin/service icinga2 onestart' in directory '/root'
<hangs forever>
load: 0.19  cmd: icinga2 70106 [sbwait] 511.95r 0.04u 0.04s 0% 19504k

Like I mentioned, I can run service icinga2 onestart, onerestart, onestop, and other options from the command line until I’m blue in the face. They never hang. It’s only when salt runs them on my behalf.

Setup

# icinga2.sls
icinga2:
  pkg.installed: []
  service.running:
    - enable: true

Steps to Reproduce Issue

salt-call -l info state.sls icinga2

Versions Report

Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.6 (default, Jan 30 2020, 01:18:54)
   python-gnupg: Not Installed
         PyYAML: 5.2
          PyZMQ: 18.1.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist:   
         locale: UTF-8
        machine: amd64
        release: 12.1-RELEASE-p2
         system: FreeBSD
        version: Not Installed

This also affected FreeBSD 11.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 19 (16 by maintainers)

Most upvoted comments

Problems solved

krionbsd on Jun 23, 2020

Severity: apocalyptic Priority: 11

😉

But seriously–thanks for looking into this.

darkpixel on Apr 15, 2020

@darkpixel as we upgraded sysutils/py-salt to 3000.3 yesterday, it would be awesome if you could test this version and this issue on it. I assume official FreeBSD 3000.3 package will be available on 12-STABLE and on HEAD tomorrow, I did not commit it into quarter builds so it won’t be available on security branches.

krionbsd on Jun 3, 2020

I can’t reproduce this failure on salt3000.3 which I’m going to port to FreeBSD in the next days. 2019.2.5 is still affected and has this issue though, stay tuned we’re working on it.

krionbsd on May 21, 2020

Same issue with PostgreSQL

silenius on Mar 12, 2020