salt: salt-call waits infinitely on apt-get of mysql server community edition
Description of Issue/Question
salt-call hangs on the following command:
[INFO ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'mysql-server'] in directory '/root'
in processes:
4 S root 30114 16398 0 80 0 - 152946 pipe_w 20:53 pts/1 00:00:01 /usr/bin/python /usr/bin/salt-call state.highstate
4 Z root 30339 30114 0 80 0 - 0 exit 20:53 pts/1 00:00:01 [apt-get] <defunct>
0 S mysql 30808 1 0 80 0 - 1112 wait 20:53 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe
0 S mysql 31598 30808 0 80 0 - 1004891 poll_s 20:53 ? 00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/storage/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/mysqld.log --open-files-limit=10000 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
in strace:
root@db02:/storage# strace -p 28116
Process 28116 attached
read(57,
in lsof:
salt-call 28116 root 57r FIFO 0,9 0t0 56123 pipe
Setup
root@infra01:/srv/salt/mysql# cat server.sls
{% from 'macros/macros.jinja' import get_local with context -%}
{% set root_password = get_local('mysql:root_creds:password', '') -%}
include:
- scripts
- zabbix.agent
- apparmor
mysql_repo:
pkgrepo.managed:
- names:
- deb http://repo.mysql.com/apt/ubuntu/ trusty mysql-5.7
- file: /etc/apt/sources.list.d/mysql.list
- keyserver: pgp.mit.edu
- keyid: 5072E1F5
- enabled: True
- refresh_db: True
mysql_group:
group.present:
- name: mysql
- system: True
mysql_user:
user.present:
- name: mysql
- groups:
- mysql
- system: True
- require:
- group: mysql
mysql_mount_dir:
file.directory:
- name: /storage
- user: root
- group: root
- mode: 755
mysql_data_dir:
file.directory:
- name: /storage/mysql
- user: mysql
- group: mysql
- mode: 755
- require:
- user: mysql_user
mysql_etc_dir:
file.directory:
- name: /etc/mysql
- user: root
- group: root
- mode: 755
mysql_config_file:
file.managed:
- name: /etc/mysql/my.cnf
- source: salt://mysql/server/my.cnf
- user: root
- group: root
- mode: 644
- require:
- file: mysql_etc_dir
mysql_conf.d_dir:
file.directory:
- name: /etc/mysql/conf.d
- user: root
- group: root
- mode: 755
- require:
- file: mysql_etc_dir
mysql_safe_config_file:
file.managed:
- name: /etc/mysql/conf.d/mysqld_safe_syslog.cnf
- source: salt://mysql/server/mysqld_safe_syslog.cnf
- user: root
- group: root
- mode: 644
- require:
- file: mysql_conf.d_dir
mysql_tuning_config_file:
file.managed:
- name: /etc/mysql/conf.d/mysqld_tuning.cnf
- source: salt://mysql/server/mysqld_tuning.cnf
- user: root
- group: root
- mode: 644
- require:
- file: mysql_conf.d_dir
mysql_repl_config_file:
file.managed:
- name: /etc/mysql/conf.d/mysqld_replication.cnf
- source: salt://mysql/server/mysqld_replication.cnf
- user: root
- group: root
- mode: 644
- require:
- file: mysql_conf.d_dir
mysql_apparmor_config:
file.managed:
- name: /etc/apparmor.d/usr.sbin.mysqld
- source: salt://mysql/server/usr.sbin.mysqld.apparmor
- user: root
- group: root
- mode: 644
- watch_in:
- service: apparmor_service
mysql_server_debconf:
debconf.set:
- name: mysql-community-server
- data:
'mysql-community-server/root-pass': {'type': 'password', 'value': '{{ root_password }}'}
'mysql-community-server/re-root-pass': {'type': 'password', 'value': '{{ root_password }}'}
mysql_server_packages:
pkg.installed:
- names:
- mysql-server
- python-mysqldb
- require:
- debconf: mysql_server_debconf
- user: mysql_user
- file: mysql_data_dir
- file: mysql_config_file
- file: mysql_safe_config_file
- file: mysql_tuning_config_file
- file: mysql_repl_config_file
- file: mysql_apparmor_config
Steps to Reproduce Issue
[INFO ] Running state [mysql-server] at time 20:53:22.501370
[INFO ] Executing state pkg.installed for mysql-server
[INFO ] Executing command 'apt-get -q update' in directory '/root'
[INFO ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'mysql-server'] in directory '/root'
Hangs forever, the install of the package is finished in the background, the DB is initialized. apt-get is a zombie.
Versions Report
root@db02:/storage# salt-call --versions-report
Salt Version:
Salt: 2015.8.10
Dependency Versions:
Jinja2: 2.7.2
M2Crypto: Not Installed
Mako: 0.9.1
PyYAML: 3.10
PyZMQ: 14.0.1
Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
RAET: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.4
cffi: Not Installed
cherrypy: Not Installed
dateutil: 1.5
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
libgit2: Not Installed
libnacl: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.3.0
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
python-gnupg: Not Installed
smmap: Not Installed
timelib: Not Installed
System Versions:
dist: Ubuntu 14.04 trusty
machine: x86_64
release: 3.19.0-59-generic
system: Ubuntu 14.04 trusty
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 23 (16 by maintainers)
Hey all, Been struggling with the issue here when trying to start a process via a custom service file on Suse 11. Salt was hanging on
service.running
state where my start code was/sbin/startproc $BIN
. I finally solved it using the -q flag on startproc/sbin/startproc -q $BIN
.I think the root cause for me was that the binary I am starting keeps a STDOUT session open and even though doing a manual
service BIN start
was exiting correctly, calling the state via salt was not.To make myself explicit: services which behave as described above (a process terminates but passes its descriptors to other processes before exiting) may indeed not be behaving as nice citizens, but the bug lies entirely in SaltStack.
When SaltStack spawns some process (i.e. runs a command) and creates a pipe to its stdout and stderr, SaltStack must establish a SIGCHLD handler. When the spawned process terminates, any I/O on such pipes must be terminated (even if the pipe is still open on the other side). Promptly reclaiming (waitpid) the exit status of a terminated child process also avoids having zombies lying around.
See [https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=214643] [Bug 214643] net-mgmt/telegraf: service (re)start hangs when scripted: missing option -f with daemon(8)
I expect we are seeing the same issue here.
To quote the essential summary from that PR:
On the SaltStack side I suppose the workaround is to provide an option to the cmd module to avoid it connecting a pipe to stdout and stderr of the spawned process.