salt: [BUG] Salt exit with bound method Client.__del__ of

Description After upgrading salt master and minion to 2019.2.4, we started getting an error We have upgraded due to CVE issue

ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored

This error occurs after several execution of salt run. When we restart the salt-master then it goes for a while, but after few hours it starts appearing again

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info).

Steps to Reproduce the behavior (Include debug logs if possible and relevant)

salt 'target' cmd.run 'ls -l /var' -l debug

[DEBUG   ] .Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client
[DEBUG   ] return event: {'example1.com': {u'failed': True}}
[DEBUG   ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO    ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG   ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 None
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG   ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO    ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG   ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None

[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored

Warning on minion log:

2020-05-05 17:00:15,248 [salt.utils.parsers:1071][WARNING ][1844] Minion received a SIGTERM. Exiting.
2020-05-05 17:02:00,540 [salt.utils.parsers:1071][WARNING ][25113] Minion received a SIGTERM. Exiting.
2020-05-06 01:07:57,521 [salt.utils.parsers:1071][WARNING ][25358] Minion received a SIGTERM. Exiting.

For simple salt.run it occurs after 10-15 continuous salt call from master

Expected behavior Error must go, instead it should print expected output

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
salt --versions-report
Salt Version:
           Salt: 2019.2.4

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.26.4
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: 3.9.7
         pygit2: 0.26.4
         Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
   python-gnupg: 0.4.4
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.6 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.6

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Comments: 24 (14 by maintainers)

Most upvoted comments

Agreed. It should definitely be a context manager, or even persisted.

Yeah actually, I’ve been trying to get my rewrite of the etcd returner merged into master since saltstack was split up into the “develop” branch like 2 years ago (#51363, with another addition in PR #55186).

Right now it’s stuck in the “Merge ports into master” project, but once that finaally gets merged into the “master” branch. I’d be willing to re-implement the logic for the etcd returner to use the etcd3 api.