salt: [3007.x][BUG] Non-root users can not call functions after upgrade to 3007.0

Description After upgrading to 3007.0, non-root users in the salt group (and in publisher_acl) can not start jobs. Permissions to ipc prevent access and are automatically reset.

Error messages:

lee@host:~$ salt '*.*' test.ping     
[WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds -   File "/usr/bin/salt", line 11, in <module>
    sys.exit(salt_main())

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 532, in salt_main
    client.run()

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/salt.py", line 192, in run
    for full_ret in cmd_func(**kwargs):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 816, in cmd_cli
    self.pub_data = self.run_job(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 388, in run_job
    pub_data = self.pub(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/__init__.py", line 1905, in pub
    if listen and not self.event.connect_pub(timeout=timeout):

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
    self.subscriber = salt.utils.asynchronous.SyncWrapper(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in __init__
    self.obj = cls(*args, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
    return publish_client(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
    return salt.transport.tcp.PublishClient(

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 219, in __init__
    super().__init__(opts, io_loop, **kwargs)

  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in __init__
    super().__init__()

Authentication error occurred.

Group membership and permissions:

lee@host:~$ groups
lee salt
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw------- 1 salt salt 0 Mar 12 09:53 workers.ipc

Setting permissions:

lee@host:~$ sudo chmod g+rw /var/run/salt/master/*
lee@host:~$ ls -l /var/run/salt/master/           
total 0
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pub.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 master_event_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 publish_pull.ipc
srw-rw---- 1 salt salt 0 Mar 12 09:53 workers.ipc
lee@host:~$ salt '*.*' test.ping
Authentication error occurred.

Logs for salt master now have:

[WARNING ] Authentication failure of type "user" occurred.

Restarting salt master:

lee@host:~$ sudo service salt-master restart
lee@host:~$ ls -l /var/run/salt/master/
total 0
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pub.ipc
srw------- 1 salt salt 0 Mar 14 10:30 master_event_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 publish_pull.ipc
srw------- 1 salt salt 0 Mar 14 10:30 workers.ipc

The log from the restart has the following line:

[ERROR   ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None

Setup Contents of /etc/salt/master.d/auth.conf:

publisher_acl:
  lee:
    - .*
  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior Follow instructions at https://docs.saltproject.io/salt/user-guide/en/latest/topics/security.html#publisher-acls to set up publisher-acl and directory permissions.

Expected behavior

Non-root user can start jobs such as test.ping and permissions to do so are not reset when (re)starting the salt-master service. Was working before upgrade to 3007.0.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.5 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 12.5 bookworm

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 29 (8 by maintainers)

Commits related to this issue

Most upvoted comments

I ran into this exact issue after upgrading from salt-master 3006.7 to 3007.0. I think I may have stumbled upon a workaround that solved the problem for me:

I did not have the python-is-python3 package installed. This made it so python commands would run as python3. Previously only python3 commands would run.

On Debian/Ubuntu: sudo apt install python-is-python3

This change seems to have fixed the issue. I no longer see the TCP Publish Client error anymore. I can now run salt '*' test.ping and other commands fine.

@lee-harmonic - does this work for you?

@roy3861 I would try Salt 3006.10, not quite the same as 2007, comms is different underneath, and 3007 has some new features, but trying 3006.10 should help identify non-root issues.

Did anyone try to set in the master config file:

ipc_mode: tcp

That COULD be a workaround for the domain socket files permissions issue(but potentially slows down the communications)

@Yarakson

Does the minion also need salt:salt?:

probably, just trial and error until it works, that’s what I did

@Yarakson

so i’ve done all the permission and still not able to ping to the minion02 from the master. Every Machine is running

altinsher@salt-master:~$ sudo salt ‘minion02’ test.ping [WARNING ] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError(‘Stream is closed’), will reconnect in 1 seconds - File “/usr/bin/salt”, line 11, in sys.exit(salt_main())

Check the permissions again. Be aware it resets the perms again whenever the salt master service is restarted.

This fixes it for me (note the different paths /var/log and /var/run

sudo chmod g+rw /var/log/salt/master
sudo chmod g+rw /var/run/salt/master/*.ipc

is this the same bug&?

2024-09-04 13:20:10,118 [salt.utils.parsers:1062][WARNING ][16404] Master received a SIGTERM. Exiting. 2024-09-04 13:20:20,103 [salt.config :2034][DEBUG ][76631] Reading configuration from /etc/salt/master 2024-09-04 13:20:20,151 [salt.config :2197][DEBUG ][76631] Including configuration from ‘/etc/salt/master.d/reactor.conf’ 2024-09-04 13:20:20,151 [salt.config :2034][DEBUG ][76631] Reading configuration from /etc/salt/master.d/reactor.conf 2024-09-04 13:20:20,300 [salt.utils.verify:599 ][WARNING ][76631] Insecure logging configuration detected! Sensitive data may be logged. 2024-09-04 13:20:42,109 [salt.utils.parsers:1062][WARNING ][76631] Master received a SIGINT. Exiting. 2024-09-04 13:23:55,086 [salt.config :2034][DEBUG ][79193] Reading configuration from /etc/salt/master 2024-09-04 13:23:55,135 [salt.config :2197][DEBUG ][79193] Including configuration from ‘/etc/salt/master.d/reactor.conf’ 2024-09-04 13:23:55,135 [salt.config :2034][DEBUG ][79193] Reading configuration from /etc/salt/master.d/reactor.conf 2024-09-04 13:23:55,283 [salt.utils.verify:599 ][WARNING ][79193] Insecure logging configuration detected! Sensitive data may be logged. 2024-09-04 13:24:49,965 [salt.transport.tcp:312 ][WARNING ][79249] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError(‘Stream is closed’), will reconnect in 1 seconds - File “/usr/bin/salt-master”, line 11, in <module> sys.exit(salt_master())

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py”, line 86, in salt_master master.start()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/daemons.py”, line 223, in start self.master.start()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/master.py”, line 844, in start salt.engines.start_engines(self.opts, self.process_manager)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/init.py”, line 59, in start_engines proc_mgr.add_process(

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 531, in add_process process.start()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 1100, in start super().start()

File “/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py”, line 121, in start self._popen = self._Popen(self)

File “/opt/saltstack/salt/lib/python3.10/multiprocessing/context.py”, line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj)

File “/opt/saltstack/salt/lib/python3.10/multiprocessing/context.py”, line 281, in _Popen return Popen(process_obj)

File “/opt/saltstack/salt/lib/python3.10/multiprocessing/popen_fork.py”, line 19, in init self._launch(process_obj)

File “/opt/saltstack/salt/lib/python3.10/multiprocessing/popen_fork.py”, line 71, in _launch code = process_obj._bootstrap(parent_sentinel=child_r)

File “/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py”, line 314, in _bootstrap self.run()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 995, in wrapped_run_func return run_func()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/init.py”, line 104, in run self.engineself.fun

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py”, line 160, in call ret = self.loader.run(run_func, *args, **kwargs)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py”, line 1269, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py”, line 1284, in _run_as return _func_or_method(*args, **kwargs)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/engines/reactor.py”, line 31, in start salt.utils.reactor.Reactor(opts).run()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py”, line 995, in wrapped_run_func return run_func()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/reactor.py”, line 214, in run with salt.utils.event.get_event(

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 127, in get_event return MasterEvent(

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 928, in init super().init(

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 265, in init self.connect_pub()

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py”, line 323, in connect_pub self.subscriber = salt.utils.asynchronous.SyncWrapper(

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py”, line 76, in init self.obj = cls(*args, **kwargs)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py”, line 210, in ipc_publish_client return publish_client(opts, io_loop, **kwargs)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py”, line 152, in publish_client return salt.transport.tcp.PublishClient(

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py”, line 220, in init super().init(opts, io_loop, **kwargs)

File “/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py”, line 398, in init super().init()

2024-09-04 13:24:50,029 [salt.utils.parsers:1062][WARNING ][79193] Master received a SIGINT. Exiting

We figured this out, turns out that the update messes up the permissions.

Before (not working, see above)

$ /var/run/salt/master$ ls -al
total 0
drwxr-xr-x 2 salt salt 120 Jul 24 08:50 .
drwxr-xr-x 4 root root  80 Jul 24 08:39 ..
srw------- 1 salt salt   0 Jul 24 08:50 master_event_pub.ipc
srw------- 1 salt salt   0 Jul 24 08:50 master_event_pull.ipc
srw------- 1 salt salt   0 Jul 24 08:50 publish_pull.ipc
srw------- 1 salt salt   0 Jul 24 08:50 workers.ipc

after (working):

$ /var/run/salt/master$ ls -al
total 0
drwxr-xr-x 2 salt salt 120 Jul 24 08:50 .
drwxr-xr-x 4 root root  80 Jul 24 08:39 ..
srw-rw---- 1 salt salt   0 Jul 24 08:50 master_event_pub.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 master_event_pull.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 publish_pull.ipc
srw-rw---- 1 salt salt   0 Jul 24 08:50 workers.ipc

The command, for anyone in doubt:

cd /var/run/salt/master
sudo chmod g+rw *.ipc