salt: Master-of-Master doesn't catch returns from Syndic Master
We have one top-level Master to which are connected ~1,000 minions and one Salt-Syndic master, which is handling just an additional 6 minions from a separate environment. The Syndic Master can communicate fine with its own minions, but when the top-level Master sends a command to the minions connected to the Syndic Master, it doesn’t register any returns.
The following log is from the salt-syndic process on the syndic master. You can see that it receives the command from the top-level master, successfully passes it to the local master, receives the returns from the minions and claims to be forwarding the events back to the top-level master, but on the top-level master itself, these returns are never seen. The top-level master sends out the follow-up “find_job” commands, which are also received passed to the minions by the syndic, the minions reply to it, and the top-level master never gets those replies.
Top-Level Master –
Salt: 2014.7.5
Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.0.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: 0.9.1
Debian source package: 2014.7.5+ds-1ubuntu1
Salt-Syndic Master –
Salt: 2014.7.5
Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.0.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: 0.9.1
Debian source package: 2014.7.5+ds-1ubuntu1
[TRACE ] Handling payload
[DEBUG ] User root Executing syndic command test.ping with jid 20150430123709158622
[DEBUG ] Command details: {'tgt_type': 'glob', 'jid': '20150430123709158622', 'tgt': 'SEC*', 'ret': '', 'to': 14, 'user': 'root', 'arg': [], 'fun': 'test.ping'}
[TRACE ] Polling timeout: 1.000000
[TRACE ] Got event 20150430123709158622
[TRACE ] Got event new_job
[TRACE ] Got event salt/job/20150430123709158622/new
[TRACE ] Polling timeout: 0.499381
[TRACE ] Got event salt/auth
[TRACE ] Polling timeout: 0.041568
[TRACE ] Got event salt/auth
[TRACE ] Polling timeout: 0.040812
[TRACE ] Got event salt/auth
[TRACE ] Polling timeout: 0.017950
[TRACE ] Polling timeout: 0.000525
[TRACE ] Polling timeout: 0.000264
[TRACE ] Polling timeout: 0.000036
[TRACE ] Forwarding events
[DEBUG ] Decrypting the current master AES key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[TRACE ] Polling timeout: 1.000000
[TRACE ] Got event 20150430123709158622
[DEBUG ] LazyLoaded mysql.get_load
[TRACE ] Got event salt/job/20150430123709158622/ret/SEC-OEM-HTXXX12
[TRACE ] Got event 20150430123709158622
[TRACE ] Got event salt/job/20150430123709158622/ret/SEC-OEM-HTMXX12
[TRACE ] Polling timeout: 0.494357
[TRACE ] Got event 20150430123709158622
[TRACE ] Polling timeout: 0.456572
[TRACE ] Got event salt/job/20150430123709158622/ret/SEC-OEM-HTRXX12
[TRACE ] Polling timeout: 0.455761
[TRACE ] Forwarding events
[INFO ] Returning information for job: 20150430123709158622
[DEBUG ] Decrypting the current master AES key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[TRACE ] ret_val =
[TRACE ] Polling timeout: 1.000000
[TRACE ] Got event salt/auth
[TRACE ] Polling timeout: 0.499934
[TRACE ] Polling timeout: 0.000023
[TRACE ] Forwarding events
[DEBUG ] Decrypting the current master AES key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[TRACE ] Polling timeout: 1.000000
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 17 (9 by maintainers)
This issue does not appear to have been resolved, as I am experiencing the same problem. Please reopen this thread.