salt: Minion timeouts with 2018.3.1
Description of Issue/Question
Since I upgraded to 2018.3.1 from 2017.7.6 I have a minion in a different datacenter that is timing out when I run salt from the master (i.e. with salt vpn\* state.apply -l debug
):
vpn01.*****:
Minion did not return. [No response]
If I use salt-call
and start the highstate from the minion, it completes correctly.
timeout value was the default one (5). Now to avoid the timeout I have to put a value higher than the highstate run
There are no issues with all the other minions in the same datacenter as the master. The only one that fails is the one in a different datacenter (that was working fine till 2017.7.6)
those are the latest lines before failing:
[DEBUG ] Checking whether jid 20180614080706375333 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] return event: {'vpn01.*****': {u'failed': True}}
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] LazyLoaded no_return.output
if I run salt-run jobs.list_job 20180614080706375333
right after it fails, I can still see the job running
Versions Report
salt master:
# salt --versions-report
Salt Version:
Salt: 2018.3.1
Dependency Versions:
cffi: 1.5.2
cherrypy: 3.5.0
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.3.7
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.0
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: 0.2.4
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-128-generic
system: Linux
version: Ubuntu 16.04 xenial
salt minion:
# salt-call --versions-report
Salt Version:
Salt: 2018.3.1
Dependency Versions:
cffi: 1.5.2
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.3.7
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.0
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-128-generic
system: Linux
version: Ubuntu 16.04 xenial
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (16 by maintainers)
I backported the patch to 2018.3.1, and works for me.
I just submitted a PR that fixed this for us, can you try it out?
patch -Np1 -d /usr/lib/python2.7/dist-packages/ < <(curl -sL https://github.com/saltstack/salt/pull/48158.diff)
Thanks, Daniel