salt: [BUG] 3006.12 fails to connect to master

Description When the minion gets updated to 3006.12 on RockyLinux8 the minion will get the following error

2025-06-17 18:22:06,353 [tornado.application:640 ][ERROR   ][82693] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f4c1703e440>, <salt.ext.tornado.concurrent.Future object at 0x7f4c177f7c70>)
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 745, in _authenticate
    creds = yield self.sign_in(channel=channel)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 885, in sign_in
    ret = self.handle_signin_response(sign_in_payload, payload)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 926, in handle_signin_response
    payload["session"], self.opts["encryption_algorithm"]
KeyError: 'session'

If you downgrade to 3006.11 the issue will go away.

This is Running Master at the following version: salt-master --version salt-master 3007.1 (Chlorine)

Setup Run the following AWS RockyLinux8 AMI: aws-marketplace/Rocky-8-EC2-Base-8.9-20231119.0.x86_64-d6577ceb-8ea8-4e0e-84c6-f098fc302e82

Install Salt Minion using bootstrap: sudo sh bootstrap-salt.sh -X -P onedir 3006

Configure Minion to connect to master

Error Present on Minion

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior See Above

Expected behavior Connecting without issues

Screenshots

Image

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

MINION

[root@ip-10-2-7-43 ~]# salt-minion --versions-report
Salt Version:
          Salt: 3006.12
 
Python Version:
        Python: 3.10.17 (main, Jun  9 2025, 20:41:48) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.19.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-513.5.1.el8_9.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

MASTER

salt --versions-report
Salt Version:
          Salt: 3007.1
 
Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.10
     gitpython: 3.1.32
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: 5.0.0
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.53.1.el8_10.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 35 (15 by maintainers)

Most upvoted comments

@mattp- I have multiple concerns with your statement

  1. Salt now has [ Latest, STS, and LTS ]. So upgrade to what version to make this work?
  2. On 3007.x [STS], the latest is 3007.4 which has a dozen recent issues logged on github. So is it expected to go from one set of errors to a dozen?

How did LTS 3006.12 [LTS] go into a total broken mode?

I can spin up a new system in AWS, fresh salt master and several minions (all running 3006.13) and over time the minions disconnect. The master is also a minion and it has the same issues.

This is really bad. I’m having to go back and find a version that is stable that I can pin.

It’s not a good one, to be sure, but in a constrained environment, it may be acceptable.

On Tue, Jun 24, 2025 at 8:35 PM Lee Clemens @.***> wrote:

leeclemens left a comment (saltstack/salt#68091) https://github.com/saltstack/salt/issues/68091#issuecomment-3001790029

Workaround for me was pinning master and minions to 3006.11 and that seems to have worked.

I don’t consider not having the security patches to address these CVEs ( https://docs.saltproject.io/en/latest/topics/releases/3006.12.html) as a “workaround.” Especially if they can “cause and[sic] arbitrary command to be run on the master”

— Reply to this email directly, view it on GitHub https://github.com/saltstack/salt/issues/68091#issuecomment-3001790029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJNGIEDZDSXEFM6ZMJYWLL3FGZBJAVCNFSM6AAAAAB7RAPCIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMBRG44TAMBSHE . You are receiving this because you were mentioned.Message ID: @.***>

– LITTLE GIRL: But which cookie will you eat FIRST? COOKIE MONSTER: Me think you have misconception of cookie-eating process.

@standaloneSA https://docs.saltproject.io/salt/install-guide/en/latest/topics/salt-version-support-lifecycle.html#cve-and-critical-support for cve backporting, backward incompatible changes can be introduced (which is what happened with these changes).

you just need to upgrade your master(s) before the minions. upgrade the masters to 3006.12 first, then the minions. that should make the minion side error you see go away.