salt: [BUG] TCP transport broken on 3004.1 and relevant bugfixes
Description Salt Master on TCP transport is broken on newly released version (https://saltproject.io/security_announcements/salt-security-advisory-release/) that mitigates several CVEs, resulting in no jobs being published.
In master logs following stacktrace can be seen:
2022-03-28 20:15:02,912 [tornado.application:640 ][ERROR ][25004] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f354fda3ea0>, <sal
t.ext.tornado.concurrent.Future object at 0x7f35462c6940>)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 294, in wrapper
result = func(*args, **kwargs)
File "/usr/lib64/python3.6/types.py", line 248, in wrapped
coro = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/transport/tcp.py", line 1565, in publish_payload
payload = salt.transport.frame.frame_msg(package["payload"])
KeyError: 'payload'
Setup TCP transport on master & minion, 3004.1 master.
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
Steps to Reproduce the behavior Run any job, doesn’t get published and no results can be obtained.
Expected behavior Jobs can actually be published.
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
Salt Version:
Salt: 3004.1
Dependency Versions:
cffi: 1.9.1
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
Jinja2: 2.11.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Aug 13 2020, 07:46:32)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.0.0
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: rhel 7.9 Maipo
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.53.1.el7.x86_64
system: Linux
version: Red Hat Enterprise Linux Server 7.9 Maipo
Additional context
https://github.com/saltstack/salt/blob/v3004.1/salt/transport/tcp.py#L1564 should be package = self.pack_publish(package) (package instead of payload) - can be applied as a workaround
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 28 (13 by maintainers)
During salt-bootstrap of the latest minion version we are seeing this error, is this traceback related as it comes from the same variable?
Your error is not the one in this issue. Your issue is you are trying to run a CVE-fixed minion against a Non CVE-fixed master.
Upgrade your master.
nm. found the problem. After the upgrade my minion configuration was erased. Updated the minion file and restarted. the minion is back online now.
I am having the same issue. I have upgraded the master to node to latest. but still ‘sig’ is not part of master response.
This is the response from master.
{'load': {'ret': True}, 'enc': 'clear'}My minions with older version continue to work fine but i am unable to register any new minions.
@lukasraska I’ve confirmed this is an issue and your suggestion seems to resolve it.