salt: [3007.x][BUG] Non-root users can not call functions after upgrade to 3007.0
Description After upgrading to 3007.0, non-root users in the salt group (and in publisher_acl) can not start jobs. Permissions to ipc prevent access and are automatically reset.
Error messages:
lee@host:~$ salt '*.*' test.ping
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/usr/bin/salt", line 11, in <module>
sys.exit(salt_main())
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 532, in salt_main
client.run()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
for full_ret in cmd_func(**kwargs):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 816, in cmd_cli
self.pub_data = self.run_job(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 388, in run_job
pub_data = self.pub(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1905, in pub
if listen and not self.event.connect_pub(timeout=timeout):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
self.subscriber = salt.utils.asynchronous.SyncWrapper(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in __init__
self.obj = cls(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
return publish_client(opts, io_loop, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
return salt.transport.tcp.PublishClient(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 219, in __init__
super().__init__(opts, io_loop, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
super().__init__()
Authentication error occurred.
Group membership and permissions:
lee@host:~$ groups
lee salt
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 workers.ipc
Setting permissions:
lee@host:~$ sudo chmod g+rw /var/run/salt/master/*
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 workers.ipc
lee@host:~$ salt '*.*' test.ping
Authentication error occurred.
Logs for salt master now have:
[WARNING ] Authentication failure of type "user" occurred.
Restarting salt master:
lee@host:~$ sudo service salt-master restart
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 publish_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 workers.ipc
The log from the restart has the following line:
[ERROR ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None
Setup
Contents of /etc/salt/master.d/auth.conf:
publisher_acl:
lee:
- .*
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Follow instructions at https://docs.saltproject.io/salt/user-guide/en/latest/topics/security.html#publisher-acls to set up publisher-acl and directory permissions.
Expected behavior
Non-root user can start jobs such as test.ping and permissions to do so are not reset when (re)starting the salt-master service. Was working before upgrade to 3007.0.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: debian 12.5 bookworm
locale: utf-8
machine: x86_64
release: 6.1.0-18-amd64
system: Linux
version: Debian GNU/Linux 12.5 bookworm
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 29 (8 by maintainers)
Commits related to this issue
- Add changelog for #66228 — committed to dwoz/salt by dwoz 2 years ago
- Add changelog for #66228 — committed to saltstack/salt by dwoz 2 years ago
I ran into this exact issue after upgrading from salt-master 3006.7 to 3007.0. I think I may have stumbled upon a workaround that solved the problem for me:
I did not have the
python-is-python3package installed. This made it sopythoncommands would run aspython3. Previously onlypython3commands would run.On Debian/Ubuntu:
sudo apt install python-is-python3This change seems to have fixed the issue. I no longer see the TCP Publish Client error anymore. I can now run
salt '*' test.pingand other commands fine.@lee-harmonic - does this work for you?
@roy3861 I would try Salt 3006.10, not quite the same as 2007, comms is different underneath, and 3007 has some new features, but trying 3006.10 should help identify non-root issues.
Did anyone try to set in the master config file:
That COULD be a workaround for the domain socket files permissions issue(but potentially slows down the communications)
@Yarakson
probably, just trial and error until it works, that’s what I did
@Yarakson
Check the permissions again. Be aware it resets the perms again whenever the salt master service is restarted.
This fixes it for me (note the different paths
/var/logand/var/runis this the same bug&?
2024-09-04 13:20:10,118 [salt.utils.parsers:1062][WARNING ][16404] Master received a SIGTERM. Exiting. 2024-09-04 13:20:20,103 [salt.config :2034][DEBUG ][76631] Reading configuration from /etc/salt/master 2024-09-04 13:20:20,151 [salt.config :2197][DEBUG ][76631] Including configuration from ‘/etc/salt/master.d/reactor.conf’ 2024-09-04 13:20:20,151 [salt.config :2034][DEBUG ][76631] Reading configuration from /etc/salt/master.d/reactor.conf 2024-09-04 13:20:20,300 [salt.utils.verify:599 ][WARNING ][76631] Insecure logging configuration detected! Sensitive data may be logged. 2024-09-04 13:20:42,109 [salt.utils.parsers:1062][WARNING ][76631] Master received a SIGINT. Exiting. 2024-09-04 13:23:55,086 [salt.config :2034][DEBUG ][79193] Reading configuration from /etc/salt/master 2024-09-04 13:23:55,135 [salt.config :2197][DEBUG ][79193] Including configuration from ‘/etc/salt/master.d/reactor.conf’ 2024-09-04 13:23:55,135 [salt.config :2034][DEBUG ][79193] Reading configuration from /etc/salt/master.d/reactor.conf 2024-09-04 13:23:55,283 [salt.utils.verify:599 ][WARNING ][79193] Insecure logging configuration detected! Sensitive data may be logged. 2024-09-04 13:24:49,965 [salt.transport.tcp:312 ][WARNING ][79249] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError(‘Stream is closed’), will reconnect in 1 seconds - File “/usr/bin/salt-master”, line 11, in <module> sys.exit(salt_master())
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py”, line 86, in salt_master master.start()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/daemons.py”, line 223, in start self.master.start()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py”, line 844, in start salt.engines.start_engines(self.opts, self.process_manager)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/init.py”, line 59, in start_engines proc_mgr.add_process(
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 531, in add_process process.start()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 1100, in start super().start()
File “/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py”, line 121, in start self._popen = self._Popen(self)
File “/opt/saltstack/salt/lib/python3.10/multiprocessing/context.py”, line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj)
File “/opt/saltstack/salt/lib/python3.10/multiprocessing/context.py”, line 281, in _Popen return Popen(process_obj)
File “/opt/saltstack/salt/lib/python3.10/multiprocessing/popen_fork.py”, line 19, in init self._launch(process_obj)
File “/opt/saltstack/salt/lib/python3.10/multiprocessing/popen_fork.py”, line 71, in _launch code = process_obj._bootstrap(parent_sentinel=child_r)
File “/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py”, line 314, in _bootstrap self.run()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 995, in wrapped_run_func return run_func()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/init.py”, line 104, in run self.engineself.fun
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py”, line 160, in call ret = self.loader.run(run_func, *args, **kwargs)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py”, line 1269, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py”, line 1284, in _run_as return _func_or_method(*args, **kwargs)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/reactor.py”, line 31, in start salt.utils.reactor.Reactor(opts).run()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 995, in wrapped_run_func return run_func()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/reactor.py”, line 214, in run with salt.utils.event.get_event(
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 127, in get_event return MasterEvent(
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 928, in init super().init(
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 265, in init self.connect_pub()
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 323, in connect_pub self.subscriber = salt.utils.asynchronous.SyncWrapper(
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py”, line 76, in init self.obj = cls(*args, **kwargs)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py”, line 210, in ipc_publish_client return publish_client(opts, io_loop, **kwargs)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py”, line 152, in publish_client return salt.transport.tcp.PublishClient(
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py”, line 220, in init super().init(opts, io_loop, **kwargs)
File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py”, line 398, in init super().init()
2024-09-04 13:24:50,029 [salt.utils.parsers:1062][WARNING ][79193] Master received a SIGINT. Exiting
We figured this out, turns out that the update messes up the permissions.
Before (not working, see above)
after (working):
The command, for anyone in doubt: