salt: close_fds=True causes cmd.run very slow
Description of Issue/Question
salt is running very slow on my system. It takes 25 seconds to run a test.ping. after some cleanup on the system, I reduced it to 12 seconds, which, is still quite slow. So I start tracking down the code. And I found the the following code in /usr/local/lib/python2.7/site-packages/salt/utils/timed_subprocess.py is causing the problem.
if self.timeout and not isinstance(self.timeout, (int, float)):
raise salt.exceptions.TimedProcTimeoutError('Error: timeout {0} must be a number'.format(self.timeout))
try:
self.process = subprocess.Popen(args, **kwargs)
except TypeError:
It takes half a second to run subprocess.Popen, which is insane. But when I run subprocess.Popen in a separate test script, it takes only 70ms
time python -c "import subprocess;subprocess.Popen(['/sbin/sysctl', '-n', 'hw.ncpu'])"
24
real 0m0.072s
user 0m0.040s
sys 0m0.039s
By adding some debug and tracing code finally I found that it was ‘close_fds’ in **kargs that caused it to run slow. If I changed close_fds to False, it will run very fast.
14:33:11,616 [TRACE ] ++++ timed_proc 0 ['/bin/kenv', 'smbios.system.product'] kwargs: {'shell': False, 'env': {'LC_NUMERIC': 'C', 'LC_MEASUREMENT': 'C', 'LC_CTYPE': 'C', 'LOGNAME': 'root', 'USER': 'root', 'PROMPT_COMMAND': '__bp_precmd_invoke_cmd; echo -ne "\\033]0;[delish]\\007"; __bp_interactive_mode', 'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin', 'LC_PAPER': 'C', 'HOME': '/root', 'PS1': '\\[\\033[1;5;31m\\]\\u@\\[\\033[1;0;31m\\]\\h\\[\\033[0;0;32m\\] \\w #\\[\\033[0m\\] ', 'PORTSDIR': '/var/poudriere/ports/default', 'LANG': 'en_AU.UTF-8', 'TERM': 'xterm-256color', 'SHELL': '/usr/local/bin/bash', 'TZ': 'Australia/Sydney', 'SHLVL': '1', 'EDITOR': 'nano', 'ITERM_ORIG_PS1': '\\[\\033[1;5;31m\\]\\u@\\[\\033[1;0;31m\\]\\h\\[\\033[0;0;32m\\] \\w #\\[\\033[0m\\] ', 'SUDO_USER': 'tao', 'USERNAME': 'root', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'SUDO_UID': '1051', '_': '/usr/local/bin/salt-call', 'SUDO_COMMAND': '/usr/local/bin/bash', 'SUDO_GID': '20', 'LC_MESSAGES': 'C', 'OLDPWD': '/usr/local/lib/python2.7/site-packages/salt', 'LC_TELEPHONE': 'C', 'HISTCONTROL': '', 'LC_MONETARY': 'C', 'PWD': '/usr/local/lib/python2.7/site-packages/salt/utils', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'LC_TIME': 'C', 'PAGER': 'more', 'LC_COLLATE': 'C', 'ITERM_PREV_PS1': '\\[\x1b]133;D;$?\x07\x1b]133;A\x07\\]\\[\\033[1;5;31m\\]\\u@\\[\\033[1;0;31m\\]\\h\\[\\033[0;0;32m\\] \\w #\\[\\033[0m\\] \\[\x1b]133;B\x07\\]'}, 'close_fds': True, 'stdout': -1, 'stderr': -2, 'cwd': '/root'}
14:33:12,091 [TRACE ] ++++ timed_proc 1
My question is: Why would we need to set close_fds to True? If I set it to False, will it cause any problems?
Setup
(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)
Steps to Reproduce Issue
(Include debug logs if possible and relevant.)
Versions Report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version: Salt: 2017.7.2
Dependency Versions: cffi: 1.7.0 cherrypy: unknown dateutil: 2.6.1 docker-py: Not Installed gitdb: 2.0.2 gitpython: 2.1.5 ioflo: Not Installed Jinja2: 2.9.5 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.7 mysql-python: 1.2.5 pycparser: 2.10 pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.14 (default, Oct 20 2017, 17:34:55) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 16.0.3 RAET: Not Installed smmap: 2.0.3 timelib: Not Installed Tornado: 4.5.2 ZMQ: 4.2.2
System Versions: dist: locale: UTF-8 machine: amd64 release: 11.1-RELEASE-p1 system: FreeBSD version: Not Installed
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 22 (18 by maintainers)
During the last days I’ve been chasing a performance issue that hit SUSE CaaS Platform.
The problem
We are using salt to deploy and manage our kubernetes cluster. We have a UI that talks with salt by using salt-api. Our salt-master is configured to create 20 workers.
We are using salt-2016.11.4 with some backported patches. This is the same version used by the SUSE Manager folks. We are also using python 2.7.13.
Once the meltdown and spectre patches got merged into SUSE CaaS we started to experience lots of timeout issues inside of our CI system. The first step of our CI creates a bunch of vanilla VMs and then bootstrap a kubernetes cluster over them using our UI. This is entirely automated using a headless browser, hence it’s faster than a human.
The meltdown and spectre patches caused a massive slowdown of the workers bootstrap. By the time the UI was up and running there was still no salt-master worker available to handle the requests done by the headless browser test suite.
Debugging
Looking at the salt-master logs I figured out it took around 2 minutes to get the first salt-master worker ready to handle incoming connections.
Running strace against the salt-master lead to an interesting discovery: I found around 5 million of “close()” system calls on invalid file descriptors in a couple minutes. The workers weren’t doing much else than these calls, they comprise 99.98% of syscalls captured in the strace. These fd numbers were all incremental.
Applications that have high syscall rates suffer the most because of meltdown fixes.
Each salt-master worker is going through the same initialization code that reads configuration files and collects facts about the machine (in this case a container) running the salt-master. Some of these facts are collected by invoking external programs.
The external programs are invoked using subprocess.Popen. Salt sets the
close_fdstoTrueon Linux systems.Python 2 implements
close_fdsby invoking the os.closerange(fd_low, fd_high) function which closes all the file descriptors fromfd_lowtofd_highignoring all the errors (doc).In this case
fd_highis obtaining by getting the value of _SC_OPEN_MAXOn our test system the value of
_SC_OPEN_MAXwas1048576. That causes python to close a lot of file descriptors that were not existing at all, leading to the generation of all the failed close()s calls I saw inside of the strace.The
_SC_OPEN_MAXvalue can be tuned by setting the value ofnofilevia ulimit. The solution to our salt-master problem consists of reducing the maximum number of file descriptor a process can open. By doing that the pythonos.closerange()will have to iterate over a smaller number of file descriptors and will generate fewerclose()syscalls.That allowed us to bring the bootstrap time of the first salt-master worker from 2 minutes down to 12 seconds.
python2 vs python3
The situation gets better with python3, according to this pep:
Translated, with python 3.3 the
closerange()function won’t generate all the errored close()s syscalls because it will focus on closing only the actually open file descriptors.Right now we are using python2, but we will soon switch to python3. That will allow us to drop the ulimit workaround.
I also know @isbm is working on a patch for salt, he just referenced this issue from his PR.