salt: salt-call waits infinitely on apt-get of mysql server community edition

Description of Issue/Question

salt-call hangs on the following command:

[INFO    ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'mysql-server'] in directory '/root'

in processes:

4 S root     30114 16398  0  80   0 - 152946 pipe_w 20:53 pts/1   00:00:01 /usr/bin/python /usr/bin/salt-call state.highstate
4 Z root     30339 30114  0  80   0 -     0 exit   20:53 pts/1    00:00:01 [apt-get] <defunct>
0 S mysql    30808     1  0  80   0 -  1112 wait   20:53 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe
0 S mysql    31598 30808  0  80   0 - 1004891 poll_s 20:53 ?      00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/storage/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/mysqld.log --open-files-limit=10000 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

in strace:

root@db02:/storage# strace -p 28116
Process 28116 attached
read(57,

in lsof:

salt-call 28116 root   57r  FIFO    0,9      0t0  56123 pipe

Setup

root@infra01:/srv/salt/mysql# cat server.sls
{% from 'macros/macros.jinja' import get_local with context -%}
{% set root_password = get_local('mysql:root_creds:password', '') -%}

include:
  - scripts
  - zabbix.agent
  - apparmor

mysql_repo:
  pkgrepo.managed:
    - names:
      - deb http://repo.mysql.com/apt/ubuntu/ trusty mysql-5.7
    - file: /etc/apt/sources.list.d/mysql.list
    - keyserver: pgp.mit.edu
    - keyid: 5072E1F5
    - enabled: True
    - refresh_db: True

mysql_group:
  group.present:
    - name: mysql
    - system: True

mysql_user:
  user.present:
    - name: mysql
    - groups:
      - mysql
    - system: True
    - require:
      - group: mysql

mysql_mount_dir:
  file.directory:
    - name: /storage
    - user: root
    - group: root
    - mode: 755

mysql_data_dir:
  file.directory:
    - name: /storage/mysql
    - user: mysql
    - group: mysql
    - mode: 755
    - require:
      - user: mysql_user

mysql_etc_dir:
  file.directory:
    - name: /etc/mysql
    - user: root
    - group: root
    - mode: 755

mysql_config_file:
  file.managed:
    - name: /etc/mysql/my.cnf
    - source: salt://mysql/server/my.cnf
    - user: root
    - group: root
    - mode: 644
    - require:
      - file: mysql_etc_dir

mysql_conf.d_dir:
  file.directory:
    - name: /etc/mysql/conf.d
    - user: root
    - group: root
    - mode: 755
    - require:
      - file: mysql_etc_dir

mysql_safe_config_file:
  file.managed:
    - name: /etc/mysql/conf.d/mysqld_safe_syslog.cnf
    - source: salt://mysql/server/mysqld_safe_syslog.cnf
    - user: root
    - group: root
    - mode: 644
    - require:
      - file: mysql_conf.d_dir

mysql_tuning_config_file:
  file.managed:
    - name: /etc/mysql/conf.d/mysqld_tuning.cnf
    - source: salt://mysql/server/mysqld_tuning.cnf
    - user: root
    - group: root
    - mode: 644
    - require:
      - file: mysql_conf.d_dir

mysql_repl_config_file:
  file.managed:
    - name: /etc/mysql/conf.d/mysqld_replication.cnf
    - source: salt://mysql/server/mysqld_replication.cnf
    - user: root
    - group: root
    - mode: 644
    - require:
      - file: mysql_conf.d_dir

mysql_apparmor_config:
  file.managed:
    - name: /etc/apparmor.d/usr.sbin.mysqld
    - source: salt://mysql/server/usr.sbin.mysqld.apparmor
    - user: root
    - group: root
    - mode: 644
    - watch_in:
      - service: apparmor_service

mysql_server_debconf:
  debconf.set:
    - name: mysql-community-server
    - data:
        'mysql-community-server/root-pass': {'type': 'password', 'value': '{{ root_password }}'}
        'mysql-community-server/re-root-pass': {'type': 'password', 'value': '{{ root_password }}'}

mysql_server_packages:
  pkg.installed:
    - names:
      - mysql-server
      - python-mysqldb
    - require:
      - debconf: mysql_server_debconf
      - user: mysql_user
      - file: mysql_data_dir
      - file: mysql_config_file
      - file: mysql_safe_config_file
      - file: mysql_tuning_config_file
      - file: mysql_repl_config_file
      - file: mysql_apparmor_config

Steps to Reproduce Issue

[INFO    ] Running state [mysql-server] at time 20:53:22.501370
[INFO    ] Executing state pkg.installed for mysql-server
[INFO    ] Executing command 'apt-get -q update' in directory '/root'
[INFO    ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'mysql-server'] in directory '/root'

Hangs forever, the install of the package is finished in the background, the DB is initialized. apt-get is a zombie.

Versions Report

root@db02:/storage# salt-call --versions-report
Salt Version:
           Salt: 2015.8.10

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: 0.9.1
         PyYAML: 3.10
          PyZMQ: 14.0.1
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.19.0-59-generic
         system: Ubuntu 14.04 trusty

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 23 (16 by maintainers)

Most upvoted comments

Hey all, Been struggling with the issue here when trying to start a process via a custom service file on Suse 11. Salt was hanging on service.running state where my start code was /sbin/startproc $BIN. I finally solved it using the -q flag on startproc /sbin/startproc -q $BIN.

I think the root cause for me was that the binary I am starting keeps a STDOUT session open and even though doing a manual service BIN start was exiting correctly, calling the state via salt was not.

To make myself explicit: services which behave as described above (a process terminates but passes its descriptors to other processes before exiting) may indeed not be behaving as nice citizens, but the bug lies entirely in SaltStack.

When SaltStack spawns some process (i.e. runs a command) and creates a pipe to its stdout and stderr, SaltStack must establish a SIGCHLD handler. When the spawned process terminates, any I/O on such pipes must be terminated (even if the pipe is still open on the other side). Promptly reclaiming (waitpid) the exit status of a terminated child process also avoids having zombies lying around.

See [https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214643] [Bug 214643] net-mgmt/telegraf: service (re)start hangs when scripted: missing option -f with daemon(8)

I expect we are seeing the same issue here.

To quote the essential summary from that PR:

I came across this issue when trying to manage a service ‘telegraf’ with SaltStack (i.e. sysutils/py-salt). I would expect the same problem can occur under other management tools like Puppet/Chef/Ansible.

What happens is that when a salt-minion tries to start or restart a telegraf process by spawning a command: ‘service telegraf restart’, the process just hangs, waiting for a restart to happen, even though the telegraf process does restart successfully and is again running underneath. […] $ service telegraf restart | cat (hangs) […] So what happens is:

  • The process executing the daemon(8) program (as started by the ‘service’ command) has its stdout directed to a pipe;
  • The daemon(8) program disassociates from a controlling terminal but DOES NOT close its stdout, and then forks;
  • The forked daemon(8) subprocess inherits the open fd to a pipe;
  • The parent daemon(8) then exits, but the child daemon(8) still has a pipe open, connected to cat(1) (or to salt-minion);
  • the child daemon(8) process then exec’s the telegraf program, which again inherits the pipe as its stdout.

On the SaltStack side I suppose the workaround is to provide an option to the cmd module to avoid it connecting a pipe to stdout and stderr of the spawned process.