salt: salt-master 2016.11.4 crashes on CentOS 7 with error "RSA key format is not supported"

Description of Issue/Question

Within the past day, package updates were applied to a CentOS 7 salt master server, upgrading salt-master and salt-minion to 2016.11.4, and installing python2-pycryptodomex (presumably as a dependency). The salt-master on this server now crashes immediately with the error “RSA key format is not supported.” No configs have been altered on this master recently. It was running 2016.11.3, and was stable for months. Now it is dead.

Setup

I copied the latest /etc/salt/master.rpmnew config file to /etc/salt/master and edited the file_roots section. I made no other changes. This is a very “vanilla” installation of Salt.

Steps to Reproduce Issue

Starting point: CentOS 7 host with Salt 2016.11.3 running

  1. Run yum update
  2. The following four updates (nothing else) were applied on April 25:
Apr 25 05:49:21 Installed: python2-pycryptodomex.x86_64 3.4.3-1.el7
Apr 25 05:49:25 Updated: salt.noarch 2016.11.4-1.el7
Apr 25 05:49:25 Updated: salt-minion.noarch 2016.11.4-1.el7
Apr 25 05:49:32 Updated: salt-master.noarch 2016.11.4-1.el
  1. Upon restart, the salt-master service crashes with the following error message:
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: Traceback (most recent call last):
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/bin/salt-master", line 22, in <module>
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: salt_master()
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 90, in salt_master
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: master.start()
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/cli/daemons.py", line 204, in start
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: super(Master, self).start()
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/utils/parsers.py", line 947, in start
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: self.prepare()
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/cli/daemons.py", line 185, in prepare
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: self.master = salt.master.Master(self.config)
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/master.py", line 399, in __init__
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: SMaster.__init__(self, opts)
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/master.py", line 119, in __init__
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: self.master_key = salt.crypt.MasterKeys(self.opts)
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 233, in __init__
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: self.key = self.__get_keys()
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 285, in __get_keys
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: key = RSA.importKey(f.read())
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib64/python2.7/site-packages/Cryptodome/PublicKey/RSA.py", line 739, in import_key
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: return _import_keyDER(der, passphrase)
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: File "/usr/lib64/python2.7/site-packages/Cryptodome/PublicKey/RSA.py", line 683, in _import_keyDER
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: raise ValueError("RSA key format is not supported")
Apr 26 02:13:55 ueb-salt-01 salt-master[2845]: ValueError: RSA key format is not supported
Apr 26 02:13:55 ueb-salt-01 systemd[1]: salt-master.service: main process exited, code=exited, status=1/FAILURE
Apr 26 02:13:55 ueb-salt-01 systemd[1]: Failed to start The Salt Master Server.
-- Subject: Unit salt-master.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit salt-master.service has failed.
-- 
-- The result is failed.

Versions Report

Salt Version:
           Salt: 2016.11.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-514.16.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 3
  • Comments: 24 (11 by maintainers)

Most upvoted comments

Hi,

I just had a similar problem with a salt-minion on CentOS 7. For me it worked if I deinstall the package python2-pycryptodomex.x86_64. It is a install dependency of salt. So you have to deinstall it with rpm -e --nodeps python2-pycryptodomex.x86_64

The following error was there before I deinstalled the package (the minion did run perfectly with 2016.11.3):

[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[CRITICAL] The Salt Master server's public key did not authenticate!
The master may need to be updated if it is a version of Salt lower than 2016.11.4, or
If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.
The master public key can be found at:
/etc/salt/pki/minion/minion_master.pub

Just for completeness this is my versions report:

[08:54 test.dev.nsynd.com .../salt/minion.d]# salt-minion --versions-report
Salt Version:
           Salt: 2016.11.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.8.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-514.2.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

The remaining mystery is why my master key generated less than one year ago is breaking pycryptodome. I will save a copy of my key, re-key, and then do some analysis on the old master.