salt: [BUG] 3004.1 broke syndication

Description After upgrading both the syndication master and other masters to 3004.1, I can no longer interact with the masters through the syndication master.

The masters have the following in /var/log/salt/syndic:

2022-03-28 14:04:36,269 [salt.pillar      :245 ][ERROR   ][16055] Pillar payload signature failed to validate.
2022-03-28 14:04:36,270 [salt.minion      :3444][ERROR   ][16055] Error while bringing up syndic for multi-syndic. Is the master at dc6-unix-salt-syndic-dev-01.nvidia.com responding?
2022-03-28 14:04:39,074 [salt.minion      :3490][ERROR   ][16055] Unable to call _fire_master on dc6-unix-salt-syndic-dev-01.nvidia.com, that syndic is not connected
2022-03-28 14:04:39,074 [salt.minion      :3501][CRITICAL][16055] Unable to call _fire_master on any masters!

Versions Report

root@dc6-unix-salt-syndic-dev-01:~# salt-master --versions-report
Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.15.0
      cherrypy: 18.6.0
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: 0.26.0
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: 0.26.2
        Python: 3.6.9 (default, Mar 15 2022, 13:55:28)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5
 
System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 4.15.0-140-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

root@dc6-unix-salt-dev-01:~# salt-syndic --versions-report
Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.14.5
      cherrypy: 18.6.1
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: 1.1.0
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.5.0
        Python: 3.6.9 (default, Mar 15 2022, 13:55:28)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5
 
Salt Extensions:
        SSEAPE: 8.4.2+4
 
System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 4.15.0-140-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 12
Comments: 24 (11 by maintainers)

Most upvoted comments

hello,

Would it be possible to have a fix-release with this bugfix ?

Thank you

Yoda-BZH on Apr 20, 2022

I used this as a dirty patch workaround for now:

--- /tmp/zeromq.py      2022-04-04 10:27:24.307557047 -0700
+++ /usr/lib/python3/dist-packages/salt/transport/zeromq.py     2022-04-04 10:29:14.054840943 -0700
@@ -382,6 +382,8 @@

         # Validate the master's signature.
         master_pubkey_path = os.path.join(self.opts["pki_dir"], "minion_master.pub")
+        if "__role" in self.opts and self.opts["__role"] == "syndic":
+            master_pubkey_path = os.path.join(self.opts["pki_dir"], "syndic_master.pub")
         if not salt.crypt.verify_signature(
             master_pubkey_path, signed_msg["data"], signed_msg["sig"]
         ):

lomeroe on Apr 4, 2022

I’m pretty sure this is due to the minion signing issue. I was seeing this error in my syndic logs:

FileNotFoundError: [Errno 2] No such file or directory: '/etc/salt/pki/minion/minion_master.pub'

I didn’t have a minion_master.pub file in that directory, because the relationship is actually a syndic<–>master. Looks like it should be possible to workaround this by creating a symlink.

cd /etc/salt/pki/minion
ln -s syndic_master.pub minion_master.pub

@ebusto @mguegan could you see if that workaround… works?

waynew on Mar 29, 2022