salt: salt-minion can't restart itself

Hi,

When I use salt service module or a watch statement on minon configuration file to restart salt-minion service, it ends up running two instances which breaks the communication between master and minion. (I recognized that PID is different after restart command)

This is required in order to be able to do mass changes on minions like editing mine configuration.

Below you can see how it behaves;

[root@mar-pre-ord-web-03 marconi]# ps -fe|grep salt
root     21507     1  7 16:43 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d

root@salt01:# salt mar-pre-ord-web-03 service.restart salt-minion

[root@mar-pre-ord-web-03 marconi]# ps -fe|grep salt
root     21583     1  1 16:44 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d
root     21601     1 25 16:44 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d

This was the sls formula, which also doesn’t work.

salt-minion:
  pkg:
    - installed
    - name: salt-minion
  file:
    - managed
    - name: /etc/salt/minion
    - template: jinja
    - source: salt://common/files/minion.jinja
    - require:
      - pkg: salt-minion
  service:
    - running
    - enable: True
    - require:
      - pkg: salt-minion
    - watch:
      - file: salt-minion

test with salt-call

[root@mar-pre-ord-web-03 marconi]# ps -fe|grep salt
root     22690     1  2 16:59 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d
root     22755  2229  0 17:00 pts/0    00:00:00 grep salt
[root@mar-pre-ord-web-03 marconi]# salt-call  service.restart salt-minion
[INFO    ] Configuration file path: /etc/salt/minion
[INFO    ] Executing command '/sbin/runlevel' in directory '/root'
[INFO    ] Executing command '/sbin/chkconfig --list' in directory '/root'
[INFO    ] Executing command '/sbin/service salt-minion restart' in directory '/root'

^C (hangs here)
Exiting gracefully on Ctrl-c
[root@mar-pre-ord-web-03 marconi]# ps -fe|grep salt
root     22788     1 19 17:00 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d
root     22821  2229  0 17:00 pts/0    00:00:00 grep salt

If I restart salt-minion on the server it works fine.

[root@mar-pre-ord-web-03 marconi]# ps -fe|grep salt
root     22253     1 18 16:56 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d

[root@mar-pre-ord-web-03 marconi]# service salt-minion restart
Stopping salt-minion daemon:                               [  OK  ]
Starting salt-minion daemon:                               [  OK  ]

[root@mar-pre-ord-web-03 marconi]# ps -fe|grep salt
root     22302     1 38 16:56 ?        00:00:00 /usr/bin/python /usr/bin/salt-minion -d

About this issue

Original URL
State: closed
Created 11 years ago
Comments: 47 (34 by maintainers)

Most upvoted comments

This should be be doable easily enough now with cmd.run_bg from #6691.

Tested this on CentOS 6 and worked fine:

salt myminion cmd.run_bg 'sleep 10; service salt-minion restart'

sjmh on Apr 15, 2016

Take a look at something we just merged a few minutes ago: https://github.com/saltstack/salt/pull/32593

cachedout on Apr 15, 2016

@vutny I can confirm that your solution works fine on Ubuntu 16.04 LTS and on Windows Server 2012 R2 with one small exception: On Windows, the complete path to salt-call has to be specified. This is strange because C:\salt is in the path grain, but maybe cmd.run_bg does not inherit this path.

Anyway, I think that the solution proposed by @vutny is the one which works best on most platforms and thus the FAQ should be updated accordingly.

I use the following logic for triggering a restart from an SLS file:

minion_config:
  # The logic for updating the minion's configuration goes here...

minion_restart:
  module.run:
    - name: cmd.run_bg
{% if grains['os'] == 'Windows' %}
    - cmd: 'C:\salt\salt-call.bat --local service.restart salt-minion'
{% else %}
    - cmd: 'salt-call --local service.restart salt-minion'
{% endif %}
    - onchanges:
      - file: minion_config

smarsching on Mar 5, 2017

@cachedout What about the changes in PR #32593? I think it’s possible to specify salt-call --local service.restart salt-minion as the minion_restart_command command and make it default. But again, not completely sure if that will work on Windows.

At least, for now we can update the FAQ for 2016.3 with better recipe.

vutny on Sep 29, 2016