salt: [BUG] 3007 master 4505 publish port failed to respond to salt commands unless restart

Description

3007 master not responding salt commands unless restarting the master. But it does answer salt-call ran from minion The timeout issue does not have log in /var/log/salt/master file with log file level set to debug. Currently trying the trace level.

log_level_logfile: trace

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • [ x] on-prem machine
  • [ x] VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • [ x] onedir packaging
  • [ x] used bootstrap to install

Steps to Reproduce the behavior $sudo salt minion-on-saltmaster test.ping

Expected behavior salt-master should answer the test.ping command quickly. In stead,it timeout.

Screenshots

  • cmd line
[me@salt01 ~]$ sudo salt salt01  test.ping
salt01:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20240328074306181473


-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 0
# of minions that did not return: 1
# of minions with errors: 0
-------------------------------------------
ERROR: Minions returned with non-zero exit code
[me@salt01 ~]$

  • and this is trace log in /etc/salt/master when a minion not responding.
2024-03-28 02:36:29,940 [salt.loader.lazy ][DEBUG   ] The functions from module 'no_return' are being loaded by dir() on the loaded module
2024-03-28 02:36:29,940 [salt.utils.lazy  ][DEBUG   ] LazyLoaded no_return.output
2024-03-28 02:36:29,941 [salt.output      ][TRACE   ] data = {'salt01': 'Minion did not return. [No response]\nThe minions may not have all finished running and any remaining minions will return upon completion. 
To look up the return data for this job later, run the following command:\n\nsalt-run jobs.lookup_jid 20240328073609814061'}
  • salt-master status
Mar 27 13:09:15 salt01 salt-master[2162920]: [ERROR   ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None
Mar 27 14:00:04 salt01 salt-master[2163029]: [ERROR   ] Requested method not exposed: _ext_nodes
Mar 27 15:00:04 isalt01 salt-master[2163274]: [ERROR   ] Requested method not exposed: _ext_nodes
<snipped>
Mar 28 03:00:04 salt01 salt-master[2163147]: [ERROR   ] Requested method not exposed: _ext_nodes
Mar 28 04:00:07 salt01 salt-master[2163282]: [ERROR   ] Requested method not exposed: _ext_nodes

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3007.0

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: rocky 8.9 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-513.18.1.el8_9.x86_64
        system: Linux
       version: Rocky Linux 8.9 Green Obsidian

Additional context Add any other context about the problem here.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 1
  • Comments: 21 (14 by maintainers)

Most upvoted comments

See also #66288 and potentially #66715 There are real issues on 3007+ with minion communications and very little in terms of response to all these.

We also had to downgrade our master to 3006.8 to resolve this issue. 3007.0 and 3007.1 did not work and we had to restart salt-master multiple times every day.

No logs are written with Loglevel WARNING. Master just refuses to connect to any minion (even itself). Salt-Call works without issues.

Hello,

Today i made update from latest 3005 to 3007.1 and i got a lot of messages in master log file: `2024-08-25 10:10:10,192 [salt.transport.tcp:1082][DEBUG ][81648] tcp stream to closed, unable to recv

2024-08-25 10:10:10,192 [salt.transport.tcp:1102][DEBUG ][81648] Subscriber at connected

2024-08-25 10:10:10,193 [salt.transport.tcp:1082][DEBUG ][81648] tcp stream to closed, unable to recv

2024-08-25 10:10:10,193 [salt.transport.tcp:1102][DEBUG ][81648] Subscriber at connected

2024-08-25 10:10:15,199 [salt.transport.tcp:1082][DEBUG ][81648] tcp stream to closed, unable to recv

2024-08-25 10:10:15,199 [salt.transport.tcp:1102][DEBUG ][81648] Subscriber at connected

2024-08-25 10:10:15,200 [salt.transport.tcp:1082][DEBUG ][81648] tcp stream to closed, unable to recv

2024-08-25 10:10:15,201 [salt.transport.tcp:1102][DEBUG ][81648] Subscriber at connected `

It seems 3007.1 is not stable and we cannot use it because communication with minions does not work (we have about 2500 minions connected to the master) After rollback to latest 3006 all works okay.

Just for information that problem still exists.

Any fix around?

I am following this PR: https://github.com/saltstack/salt/pull/66335 , not really know the details. but this issue looks like a publish port issue that 66335 trying to fix.