salt: Minion timeouts with 2018.3.1

Description of Issue/Question

Since I upgraded to 2018.3.1 from 2017.7.6 I have a minion in a different datacenter that is timing out when I run salt from the master (i.e. with salt vpn\* state.apply -l debug):

vpn01.*****:
    Minion did not return. [No response]

If I use salt-call and start the highstate from the minion, it completes correctly.

timeout value was the default one (5). Now to avoid the timeout I have to put a value higher than the highstate run

There are no issues with all the other minions in the same datacenter as the master. The only one that fails is the one in a different datacenter (that was working fine till 2017.7.6)

those are the latest lines before failing:

[DEBUG   ] Checking whether jid 20180614080706375333 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG   ] return event: {'vpn01.*****': {u'failed': True}}
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded localfs.init_kwargs
[DEBUG   ] LazyLoaded no_return.output

if I run salt-run jobs.list_job 20180614080706375333 right after it fails, I can still see the job running

Versions Report

salt master:

# salt --versions-report
Salt Version:
           Salt: 2018.3.1

Dependency Versions:
           cffi: 1.5.2
       cherrypy: 3.5.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-128-generic
         system: Linux
        version: Ubuntu 16.04 xenial

salt minion:

# salt-call --versions-report
Salt Version:
           Salt: 2018.3.1

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.3.7
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: 0.2.4
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-128-generic
         system: Linux
        version: Ubuntu 16.04 xenial

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

I backported the patch to 2018.3.1, and works for me.

I just submitted a PR that fixed this for us, can you try it out?

patch -Np1 -d /usr/lib/python2.7/dist-packages/ < <(curl -sL https://github.com/saltstack/salt/pull/48158.diff)

Thanks, Daniel