salt: using clean=True parameter in file.recurse causes python process to spin out of control

Description of Issue/Question

As the title says, I was managing the salt-minion config files with file.recurse. When I added the clean=True flag, the python process would go crazy and eat up all the cpu.

Starting the salt-minion-debug service, I would see messages like this repeated over and over

[DEBUG   ] In saltenv 'base', ** considering ** path
'c:\salt\var\cache\salt\minion\files\base\salt-minion-config\minion.d\win.conf' to resolve
'salt://salt-minion-config/minion.d/win.conf'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://salt-minion-config/minion.d/win.conf'
[DEBUG   ] No dest file found
[INFO    ] Fetching file from saltenv 'base', ** done ** 'salt-minion-config/minion.d/win.conf'
[DEBUG   ] Jinja search path: '['c:\\salt\\var\\cache\\salt\\minion\\files\\base']'
[DEBUG   ] In saltenv 'base', looking at rel_path 'salt-minion-config/minion.d/win.conf' to resolve
'salt://salt-minion-config/minion.d/win.conf'

The files in question were present on the minion, in the cached files and in the target directory, and on the master. Without the clean=True flag the python process behaves normally, and there are no errors like above.

Setup

#!pyobjects

File.recurse(
  'salt-minion-config',
  name     = 'C:/salt/conf/minion.d',
  source   = 'salt://salt-minion-config/minion.d',
  defaults = pillar('minion:defaults'),
  template = 'jinja',
  clean      = True
)

Steps to Reproduce Issue

Add/Remove “clean=True” argument

Versions Report

Salt Version:
           Salt: 2016.3.0

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 5.3.0
       dateutil: 2.5.3
          gitdb: 0.6.4
      gitpython: 2.0.2
          ioflo: 1.5.3
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.4
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 0.7.2.None
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)]
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist:
        machine: AMD64
        release: 2012ServerR2
         system: Windows
        version: 2012ServerR2 6.3.9600  Multiprocessor Free

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 17 (9 by maintainers)

Commits related to this issue

Most upvoted comments

@twangboy Based on our conversation today, I think the root of this issue is a call to minionfs’s file_list method which run’s on the master and returns unix style paths. Windows minions call while generating keep. This happens in the recurse method on 2016.11 and in the _gen_recurse_managed_files method in more recent versions. It looks like the keep list can get populated with a mix of windows style and unix style paths based on what I’m seeing.

With all this I think I am fairly close to being able to pin this down with a regression test.