salt: [BUG] 3006.12 fails to connect to master
Description When the minion gets updated to 3006.12 on RockyLinux8 the minion will get the following error
2025-06-17 18:22:06,353 [tornado.application:640 ][ERROR ][82693] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f4c1703e440>, <salt.ext.tornado.concurrent.Future object at 0x7f4c177f7c70>)
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 745, in _authenticate
creds = yield self.sign_in(channel=channel)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 885, in sign_in
ret = self.handle_signin_response(sign_in_payload, payload)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 926, in handle_signin_response
payload["session"], self.opts["encryption_algorithm"]
KeyError: 'session'
If you downgrade to 3006.11 the issue will go away.
This is Running Master at the following version: salt-master --version salt-master 3007.1 (Chlorine)
Setup Run the following AWS RockyLinux8 AMI: aws-marketplace/Rocky-8-EC2-Base-8.9-20231119.0.x86_64-d6577ceb-8ea8-4e0e-84c6-f098fc302e82
Install Salt Minion using bootstrap: sudo sh bootstrap-salt.sh -X -P onedir 3006
Configure Minion to connect to master
Error Present on Minion
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior See Above
Expected behavior Connecting without issues
Screenshots
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)MINION
[root@ip-10-2-7-43 ~]# salt-minion --versions-report
Salt Version:
Salt: 3006.12
Python Version:
Python: 3.10.17 (main, Jun 9 2025, 20:41:48) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.19.3
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.10 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.5.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.10 Green Obsidian
MASTER
salt --versions-report
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: 4.0.10
gitpython: 3.1.32
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: 5.0.0
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: rocky 8.10 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-553.53.1.el8_10.x86_64
system: Linux
version: Rocky Linux 8.10 Green Obsidian
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 35 (15 by maintainers)
@mattp- I have multiple concerns with your statement
How did LTS 3006.12 [LTS] go into a total broken mode?
I can spin up a new system in AWS, fresh salt master and several minions (all running 3006.13) and over time the minions disconnect. The master is also a minion and it has the same issues.
This is really bad. I’m having to go back and find a version that is stable that I can pin.
It’s not a good one, to be sure, but in a constrained environment, it may be acceptable.
On Tue, Jun 24, 2025 at 8:35 PM Lee Clemens @.***> wrote:
– LITTLE GIRL: But which cookie will you eat FIRST? COOKIE MONSTER: Me think you have misconception of cookie-eating process.
@standaloneSA https://docs.saltproject.io/salt/install-guide/en/latest/topics/salt-version-support-lifecycle.html#cve-and-critical-support for cve backporting, backward incompatible changes can be introduced (which is what happened with these changes).
@twangboy #68095
you just need to upgrade your master(s) before the minions. upgrade the masters to 3006.12 first, then the minions. that should make the minion side error you see go away.