salt: [BUG] Salt exit with bound method Client.__del__ of Description
After upgrading salt master and minion to 2019.2.4, we started getting an error
We have upgraded due to CVE issue
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored
This error occurs after several execution of salt run.
When we restart the salt-master then it goes for a while, but after few hours it starts appearing again
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info).
Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
salt 'target' cmd.run 'ls -l /var' -l debug
[DEBUG ] .Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client
[DEBUG ] return event: {'example1.com': {u'failed': True}}
[DEBUG ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 None
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored
Warning on minion log:
2020-05-05 17:00:15,248 [salt.utils.parsers:1071][WARNING ][1844] Minion received a SIGTERM. Exiting.
2020-05-05 17:02:00,540 [salt.utils.parsers:1071][WARNING ][25113] Minion received a SIGTERM. Exiting.
2020-05-06 01:07:57,521 [salt.utils.parsers:1071][WARNING ][25358] Minion received a SIGTERM. Exiting.
For simple salt.run it occurs after 10-15 continuous salt call from master
Expected behavior
Error must go, instead it should print expected output
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
salt --versions-report
Salt Version:
Salt: 2019.2.4
Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.26.4
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: 3.9.7
pygit2: 0.26.4
Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
python-gnupg: 0.4.4
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: redhat 7.6 Maipo
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.el7.x86_64
system: Linux
version: Red Hat Enterprise Linux Server 7.6
Additional context
Add any other context about the problem here.
Description After upgrading salt master and minion to 2019.2.4, we started getting an error We have upgraded due to CVE issue
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored
This error occurs after several execution of salt run. When we restart the salt-master then it goes for a while, but after few hours it starts appearing again
Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info).
Steps to Reproduce the behavior (Include debug logs if possible and relevant)
salt 'target' cmd.run 'ls -l /var' -l debug
[DEBUG ] .Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client
[DEBUG ] return event: {'example1.com': {u'failed': True}}
[DEBUG ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 None
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored
Warning on minion log:
2020-05-05 17:00:15,248 [salt.utils.parsers:1071][WARNING ][1844] Minion received a SIGTERM. Exiting.
2020-05-05 17:02:00,540 [salt.utils.parsers:1071][WARNING ][25113] Minion received a SIGTERM. Exiting.
2020-05-06 01:07:57,521 [salt.utils.parsers:1071][WARNING ][25358] Minion received a SIGTERM. Exiting.
For simple salt.run it occurs after 10-15 continuous salt call from master
Expected behavior Error must go, instead it should print expected output
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)salt --versions-report
Salt Version:
Salt: 2019.2.4
Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.26.4
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: 3.9.7
pygit2: 0.26.4
Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
python-gnupg: 0.4.4
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: redhat 7.6 Maipo
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.el7.x86_64
system: Linux
version: Red Hat Enterprise Linux Server 7.6
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 24 (14 by maintainers)
Agreed. It should definitely be a context manager, or even persisted.
Yeah actually, I’ve been trying to get my rewrite of the etcd returner merged into master since saltstack was split up into the “develop” branch like 2 years ago (#51363, with another addition in PR #55186).
Right now it’s stuck in the “Merge ports into master” project, but once that finaally gets merged into the “master” branch. I’d be willing to re-implement the logic for the etcd returner to use the etcd3 api.