salt: Master-of-Master doesn't catch returns from Syndic Master

We have one top-level Master to which are connected ~1,000 minions and one Salt-Syndic master, which is handling just an additional 6 minions from a separate environment. The Syndic Master can communicate fine with its own minions, but when the top-level Master sends a command to the minions connected to the Syndic Master, it doesn’t register any returns.

The following log is from the salt-syndic process on the syndic master. You can see that it receives the command from the top-level master, successfully passes it to the local master, receives the returns from the minions and claims to be forwarding the events back to the top-level master, but on the top-level master itself, these returns are never seen. The top-level master sends out the follow-up “find_job” commands, which are also received passed to the minions by the syndic, the minions reply to it, and the top-level master never gets those replies.

Top-Level Master –

                  Salt: 2014.7.5
                Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
 Debian source package: 2014.7.5+ds-1ubuntu1

Salt-Syndic Master –

                  Salt: 2014.7.5
                Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
                Jinja2: 2.7.2
              M2Crypto: 0.21.1
        msgpack-python: 0.3.0
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.10
                 ioflo: Not Installed
                 PyZMQ: 14.0.1
                  RAET: Not Installed
                   ZMQ: 4.0.4
                  Mako: 0.9.1
 Debian source package: 2014.7.5+ds-1ubuntu1
[TRACE   ] Handling payload
[DEBUG   ] User root Executing syndic command test.ping with jid 20150430123709158622
[DEBUG   ] Command details: {'tgt_type': 'glob', 'jid': '20150430123709158622', 'tgt': 'SEC*', 'ret': '', 'to': 14, 'user': 'root', 'arg': [], 'fun': 'test.ping'}
[TRACE   ] Polling timeout: 1.000000
[TRACE   ] Got event 20150430123709158622
[TRACE   ] Got event new_job
[TRACE   ] Got event salt/job/20150430123709158622/new
[TRACE   ] Polling timeout: 0.499381
[TRACE   ] Got event salt/auth
[TRACE   ] Polling timeout: 0.041568
[TRACE   ] Got event salt/auth
[TRACE   ] Polling timeout: 0.040812
[TRACE   ] Got event salt/auth
[TRACE   ] Polling timeout: 0.017950
[TRACE   ] Polling timeout: 0.000525
[TRACE   ] Polling timeout: 0.000264
[TRACE   ] Polling timeout: 0.000036
[TRACE   ] Forwarding events
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[TRACE   ] Polling timeout: 1.000000
[TRACE   ] Got event 20150430123709158622
[DEBUG   ] LazyLoaded mysql.get_load
[TRACE   ] Got event salt/job/20150430123709158622/ret/SEC-OEM-HTXXX12
[TRACE   ] Got event 20150430123709158622
[TRACE   ] Got event salt/job/20150430123709158622/ret/SEC-OEM-HTMXX12
[TRACE   ] Polling timeout: 0.494357
[TRACE   ] Got event 20150430123709158622
[TRACE   ] Polling timeout: 0.456572
[TRACE   ] Got event salt/job/20150430123709158622/ret/SEC-OEM-HTRXX12
[TRACE   ] Polling timeout: 0.455761
[TRACE   ] Forwarding events
[INFO    ] Returning information for job: 20150430123709158622
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[TRACE   ] ret_val = 
[TRACE   ] Polling timeout: 1.000000
[TRACE   ] Got event salt/auth
[TRACE   ] Polling timeout: 0.499934
[TRACE   ] Polling timeout: 0.000023
[TRACE   ] Forwarding events
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[TRACE   ] Polling timeout: 1.000000

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 17 (9 by maintainers)

Most upvoted comments

This issue does not appear to have been resolved, as I am experiencing the same problem. Please reopen this thread.