salt: win_system.join_domain fails in version 2019.2.1

Description of Issue

After upgrade from 2018.3.4 to 2019.2.1 (master & minion) windows_system.join_domain doesn’t work anymore from master. Working properly from minion with salt call. Working properly with master 2019.2.1 and minion 2018.3.4.

Setup

{% set domain_info = salt['pillar.get']('domain', {}) %}
{% if domain_info.add_to_domain %}
  join_to_{domain_info.domain_name}:
    system.join_domain:
      - name: {{ domain_info.get('domain_name') }}
      - username: {{ domain_info.get('user', 'Administrator')}}
      - password: {{ domain_info.get('password') }}
      - account_exists: {{ domain_info.get('account_exists', 'False') }}
      - restart: {{ domain_info.get('restart', 'False') }}
      {% if 'account_ou' in domain_info -%}
      - account_ou: {{ domain_info.get('account_ou') }}
      {% endif -%}
{% endif %}

Steps to Reproduce Issue

Debug from minion with salt-call

[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] Including configuration from 'c:\salt\conf\minion.d\_schedule.conf'
[DEBUG   ] Reading configuration from c:\salt\conf\minion.d\_schedule.conf
[DEBUG   ] Configuration file path: c:\salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] Including configuration from 'c:\salt\conf\minion.d\_schedule.conf'
[DEBUG   ] Reading configuration from c:\salt\conf\minion.d\_schedule.conf
[DEBUG   ] Motherboard info not available on this system
[DEBUG   ] Loading static grains from c:\salt\conf\grains
[DEBUG   ] "il-octopus.corp.projecteam.com" Not an IP address? Assuming it is a hostname.
[DEBUG   ] Master URI: tcp://1.1.1.1:4506
[DEBUG   ] Connecting to master. Attempt 1 (infinite attempts)
[DEBUG   ] "il-octopus.corp.projecteam.com" Not an IP address? Assuming it is a hostname.
[DEBUG   ] Master URI: tcp://1.1.1.1:4506
[DEBUG   ] Popen(['git', 'version'], cwd=C:\Users\Administrator, universal_newlines=False, shell=None)
[DEBUG   ] Initializing new AsyncAuth for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (6319)
[DEBUG   ] Setting zmq_reconnect_ivl to '6319ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506', 'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://1.1.1.1:4506
[DEBUG   ] Trying to connect to: tcp://1.1.1.1:4506
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG   ] Loaded minion key: c:\minion.pem
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Connecting the Minion to the Master publish port, using the URI: tcp://1.1.1.1:4505
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: c:\minion.pem
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://1.1.1.1:4506
[DEBUG   ] Trying to connect to: tcp://1.1.1.1:4506
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: c:\minion.pem
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] key: test, ret: _|-
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://1.1.1.1:4506
[DEBUG   ] Trying to connect to: tcp://1.1.1.1:4506
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Could not find file 'salt://add2domain.sls' in saltenv 'base'
[DEBUG   ] In saltenv 'base', looking at rel_path 'add2domain/init.sls' to resolve 'salt://add2domain/init.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'c:\salt\var\cache\salt\minion\files\base\add2domain\init.sls' to resolve 'salt://add2domain/init.sls'
[DEBUG   ] compile template: c:\salt\var\cache\salt\minion\files\base\add2domain\init.sls
[DEBUG   ] Jinja search path: ['c:\\salt\\var\\cache\\salt\\minion\\files\\base']
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://1.1.1.1:4506
[DEBUG   ] Trying to connect to: tcp://1.1.1.1:4506
[DEBUG   ] LazyLoaded pillar.get
[PROFILE ] Time (in seconds) to render 'c:\salt\var\cache\salt\minion\files\base\add2domain\init.sls' using 'jinja' renderer: 0.046860456466674805
[DEBUG   ] Rendered data from file: c:\salt\var\cache\salt\minion\files\base\add2domain\init.sls:


  join_to_{domain_info.domain_name}:
    system.join_domain:
      - name: my.domaim.com
      - username: adminUser
      - password: adminPassword
      - account_exists: False
      - restart: False


[DEBUG   ] Results of YAML rendering:
OrderedDict([('join_to_{domain_info.domain_name}', OrderedDict([('system.join_domain', [OrderedDict([('name', 'my.domaim.com')]), OrderedDict([('username', 'adminUser')]), OrderedDict([('password', 'adminPassword')]), OrderedDict([('account_exists', False)]), OrderedDict([('restart', False)])])]))])
[PROFILE ] Time (in seconds) to render 'c:\salt\var\cache\salt\minion\files\base\add2domain\init.sls' using 'yaml' renderer: 0.0
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] LazyLoaded system.get_computer_desc
[DEBUG   ] LazyLoaded system.join_domain
[INFO    ] Running state [my.domaim.com] at time 13:47:09.983565
[INFO    ] Executing state system.join_domain for [my.domaim.com]
[INFO    ] {'new': 'my.domaim.com', 'old': 'Workgroup'}
[INFO    ] Completed state [my.domaim.com] at time 13:47:10.811706 (duration_in_ms=828.141)
[DEBUG   ] File c:\salt\var\cache\salt\minion\accumulator\2016240778600 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command ['attrib', '-R', 'c:\\salt\\var\\cache\\salt\\minion\\sls.p'] in directory 'C:\Users\Administrator'
[DEBUG   ] output:
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('c:\\salt\\conf\\pki\\minion', 'uk1-drjob10', 'tcp://1.1.1.1:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://1.1.1.1:4506
[DEBUG   ] Trying to connect to: tcp://1.1.1.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output
local:
----------
          ID: join_to_{domain_info.domain_name}
    Function: system.join_domain
        Name: my.domaim.com
      Result: True
     Comment: Computer added to 'my.domaim.com'
              System needs to be restarted
     Started: 13:47:09.983565
    Duration: 828.141 ms
     Changes:
              ----------
              new:
                  my.domaim.com
              old:
                  Workgroup

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 828.141 ms
[DEBUG   ] Closing AsyncZeroMQReqChannel instance

Debug from master: salt ‘uk1-drjob10’ state.apply add2domain -l debug

[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/ext_pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ext_pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_servers.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_servers.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/loadbalancers.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/loadbalancers.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltMaster.domain.com
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/ext_pillar.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/ext_pillar.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_servers.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_servers.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/loadbalancers.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/loadbalancers.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: saltMaster.domain.com
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'saltMaster.domain.com_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] Closing AsyncZeroMQReqChannel instance
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/83/6af3edfcf685b7cf18133ffec1518d409fa01b7a932f03d5888a5f54a7af90/.minions.p
[DEBUG   ] get_iter_returns for jid 20191003160616528160 sent to set(['uk1-drjob10']) will timeout at 16:07:16.539631
[DEBUG   ] jid 20191003160616528160 return from uk1-drjob10
[DEBUG   ] return event: {u'uk1-drjob10': {u'jid': u'20191003160616528160', u'retcode': 2, u'ret': {u'system_|-join_to_{domain_info.domain_name}_|-uk.eu.com_|-join_domain': {u'comment': u'An exception occurred in this state: Traceback (most recent
call last):\n  File "c:\\salt\\bin\\lib\\site-packages\\wmi.py", line 1145, in __getattr__\n    return self._cached_classes
(attribute)\n  File "c:\\salt\\bin\\lib\\site-packages\\wmi.py", line 1156, in _cached_classes\n    self._classes_map[class_name] = _wmi_class (self, self._namespace.Get (class_name))\n  File "<COMObject winmgmts:>", line 3, in Get\n  File "c:\\salt\\bin\\lib\\site-packages\\win32com\\client\\dynamic.py", line 287, in _ApplyTypes_\n    result = self._oleobj_.InvokeTypes(*(dispid, LCID, wFlags, retType, argTypes) + args)\npywintypes.com_error: (-2147352567, \'Exception occurred.\', (0, \'SWbemServicesEx\', None, None, 0, -2147417848), None)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "c:\\salt\\bin\\lib\\site-packages\\salt\\state.py", line 1933, in call\n    **cdata[\'kwargs\'])\n  File "c:\\salt\\bin\\lib\\site-packages\\salt\\loader.py", line 1951, in wrapper\n    return f(*args, **kwargs)\n  File "c:\\salt\\bin\\lib\\site-packages\\salt\\states\\win_system.py", line 223, in join_domain\n    current_domain_dic = __salt__[\'system.get_domain_workgroup\']()\n  File "c:\\salt\\bin\\lib\\site-packages\\salt\\modules\\win_system.py", line 986, in get_domain_workgroup\n    for computer in conn.Win32_ComputerSystem():\n  File "c:\\salt\\bin\\lib\\site-packages\\wmi.py", line 1147, in __getattr__\n    return getattr (self._namespace, attribute)\n  File "c:\\salt\\bin\\lib\\site-packages\\win32com\\client\\dynamic.py", line 527, in __getattr__\n    raise AttributeError("%s.%s" % (self._username_, attr))\nAttributeError: winmgmts:.Win32_ComputerSystem\n', u'name': u'uk.eu.com', u'start_time': u'14:06:53.218588', u'result': False, u'duration': 15.625, u'__run_num__': 0, u'__sls__': u'add2domain', u'changes': {}, u'__id__': u'join_to_{domain_info.domain_name}'}}, u'out': u'highstate'}}
[DEBUG   ] LazyLoaded highstate.output
uk1-drjob10:
----------
          ID: join_to_{domain_info.domain_name}
    Function: system.join_domain
        Name: uk.eu.com
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "c:\salt\bin\lib\site-packages\wmi.py", line 1145, in __getattr__
                  return self._cached_classes (attribute)
                File "c:\salt\bin\lib\site-packages\wmi.py", line 1156, in _cached_classes
                  self._classes_map[class_name] = _wmi_class (self, self._namespace.Get (class_name))
                File "<COMObject winmgmts:>", line 3, in Get
                File "c:\salt\bin\lib\site-packages\win32com\client\dynamic.py", line 287, in _ApplyTypes_
                  result = self._oleobj_.InvokeTypes(*(dispid, LCID, wFlags, retType, argTypes) + args)
              pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, 'SWbemServicesEx', None, None, 0, -2147417848), None)

              During handling of the above exception, another exception occurred:

              Traceback (most recent call last):
                File "c:\salt\bin\lib\site-packages\salt\state.py", line 1933, in call
                  **cdata['kwargs'])
                File "c:\salt\bin\lib\site-packages\salt\loader.py", line 1951, in wrapper
                  return f(*args, **kwargs)
                File "c:\salt\bin\lib\site-packages\salt\states\win_system.py", line 223, in join_domain
                  current_domain_dic = __salt__['system.get_domain_workgroup']()
                File "c:\salt\bin\lib\site-packages\salt\modules\win_system.py", line 986, in get_domain_workgroup
                  for computer in conn.Win32_ComputerSystem():
                File "c:\salt\bin\lib\site-packages\wmi.py", line 1147, in __getattr__
                  return getattr (self._namespace, attribute)
                File "c:\salt\bin\lib\site-packages\win32com\client\dynamic.py", line 527, in __getattr__
                  raise AttributeError("%s.%s" % (self._username_, attr))
              AttributeError: winmgmts:.Win32_ComputerSystem
     Started: 14:06:53.218588
    Duration: 15.625 ms
     Changes:

Summary for uk1-drjob10
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:  15.625 ms
[DEBUG   ] jid 20191003160616528160 found all minions set([u'uk1-drjob10'])
[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code

Tested the module itself from master as well. same exceptions:

[root@myMaster ~]# salt 'uk1-drjob10' system.join_domain domain='my.domain.com' \
>                  username='adminUser' password='adminPassword' \
>                  account_exists=False, restart=false
uk1-drjob10:
    The minion function caused an exception: Traceback (most recent call last):
      File "c:\salt\bin\lib\site-packages\wmi.py", line 1145, in __getattr__
        return self._cached_classes (attribute)
      File "c:\salt\bin\lib\site-packages\wmi.py", line 1156, in _cached_classes
        self._classes_map[class_name] = _wmi_class (self, self._namespace.Get (class_name))
      File "<COMObject winmgmts:>", line 3, in Get
      File "c:\salt\bin\lib\site-packages\win32com\client\dynamic.py", line 287, in _ApplyTypes_
        result = self._oleobj_.InvokeTypes(*(dispid, LCID, wFlags, retType, argTypes) + args)
    pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, 'SWbemServicesEx', None, None, 0, -2147417848), None)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "c:\salt\bin\lib\site-packages\salt\minion.py", line 1663, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "c:\salt\bin\lib\site-packages\salt\executors\direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "c:\salt\bin\lib\site-packages\salt\modules\win_system.py", line 770, in join_domain
        status = get_domain_workgroup()
      File "c:\salt\bin\lib\site-packages\salt\modules\win_system.py", line 986, in get_domain_workgroup
        for computer in conn.Win32_ComputerSystem():
      File "c:\salt\bin\lib\site-packages\wmi.py", line 1147, in __getattr__
        return getattr (self._namespace, attribute)
      File "c:\salt\bin\lib\site-packages\win32com\client\dynamic.py", line 527, in __getattr__
        raise AttributeError("%s.%s" % (self._username_, attr))
    AttributeError: winmgmts:.Win32_ComputerSystem
ERROR: Minions returned with non-zero exit code

Versions Report

Salt Version: Salt: 2019.2.1

Dependency Versions: cffi: 1.12.2 cherrypy: unknown dateutil: 1.5 docker-py: Not Installed gitdb: 0.6.4 gitpython: 1.0.1 ioflo: Not Installed Jinja2: 2.7.2 libgit2: 0.26.3 libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: 2.14 pycrypto: 2.6.1 pycryptodome: 3.7.0 pygit2: 0.26.4 Python: 2.7.5 (default, Oct 30 2018, 23:45:53) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: 0.9.0 timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: centos 7.6.1810 Core locale: UTF-8 machine: x86_64 release: 3.10.0-514.26.2.el7.x86_64 system: Linux version: CentOS Linux 7.6.1810 Core