salt: using clean=True parameter in file.recurse causes python process to spin out of control
Description of Issue/Question
As the title says, I was managing the salt-minion config files with file.recurse. When I added the clean=True flag, the python process would go crazy and eat up all the cpu.
Starting the salt-minion-debug service, I would see messages like this repeated over and over
[DEBUG ] In saltenv 'base', ** considering ** path
'c:\salt\var\cache\salt\minion\files\base\salt-minion-config\minion.d\win.conf' to resolve
'salt://salt-minion-config/minion.d/win.conf'
[DEBUG ] Fetching file from saltenv 'base', ** attempting ** 'salt://salt-minion-config/minion.d/win.conf'
[DEBUG ] No dest file found
[INFO ] Fetching file from saltenv 'base', ** done ** 'salt-minion-config/minion.d/win.conf'
[DEBUG ] Jinja search path: '['c:\\salt\\var\\cache\\salt\\minion\\files\\base']'
[DEBUG ] In saltenv 'base', looking at rel_path 'salt-minion-config/minion.d/win.conf' to resolve
'salt://salt-minion-config/minion.d/win.conf'
The files in question were present on the minion, in the cached files and in the target directory, and on the master. Without the clean=True flag the python process behaves normally, and there are no errors like above.
Setup
#!pyobjects
File.recurse(
'salt-minion-config',
name = 'C:/salt/conf/minion.d',
source = 'salt://salt-minion-config/minion.d',
defaults = pillar('minion:defaults'),
template = 'jinja',
clean = True
)
Steps to Reproduce Issue
Add/Remove “clean=True” argument
Versions Report
Salt Version:
Salt: 2016.3.0
Dependency Versions:
cffi: 1.6.0
cherrypy: 5.3.0
dateutil: 2.5.3
gitdb: 0.6.4
gitpython: 2.0.2
ioflo: 1.5.3
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.4
msgpack-pure: Not Installed
msgpack-python: 0.4.7
mysql-python: 0.7.2.None
pycparser: 2.14
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)]
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: 0.2.4
Tornado: 4.3
ZMQ: 4.1.2
System Versions:
dist:
machine: AMD64
release: 2012ServerR2
system: Windows
version: 2012ServerR2 6.3.9600 Multiprocessor Free
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 17 (9 by maintainers)
Commits related to this issue
- Fix file.recurse w/ clean=True #36802 Fix an endless looping but in file.recurse when clean is set to true. This patch normalizes paths and handles the root correctly. — committed to dwoz/salt by dwoz 6 years ago
- Merge pull request #46632 from dwoz/file-recurse-36802 Fix file.recurse w/ clean=True #36802 — committed to saltstack/salt by deleted user 6 years ago
+1
@twangboy Based on our conversation today, I think the root of this issue is a call to minionfs’s file_list method which run’s on the master and returns unix style paths. Windows minions call while generating keep. This happens in the recurse method on 2016.11 and in the _gen_recurse_managed_files method in more recent versions. It looks like the
keep
list can get populated with a mix of windows style and unix style paths based on what I’m seeing.With all this I think I am fairly close to being able to pin this down with a regression test.