salt: [BUG] intermittent connection between master and minion
Description
A clear and concise description of what the bug is.
I am seeing a weird connection issue in my salt setup. there are ~30 minions registered with the master. for a few of them, master couldn’t connect to them anymore after a while. salt '*' test.ping failed with the following error message:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20230920213139507242
here are a few observations:
- restarting the salt-minion service helped but the same minion lost connection again after a while.
salt-call test.pingworks fine on minion side. other commands likesalt-call state.applyalso works fine. this indicates minion to master communication is fine but master to minion communication is not- below is the error message i found from minion log. i tried to bump up the timeout param like salt ‘*’ -t 600 test.ping . but it doesn’t help
2023-09-20 13:00:08,121 [salt.minion :2733][ERROR ][821760] Timeout encountered while sending {'cmd': '_return', 'id': 'minion', 'success': True, 'return': True, 'retcode': 0, 'jid': '20230920195941006337', 'fun': 'test.ping', 'fun_args': [], 'user': 'root', '_stamp': '2023-09-20T19:59:41.114944', 'nonce': '3b23a38761fc4e98a694448d36ac7f97'} request
does anyone have any idea what's wrong here and how to debug this issue?
Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
- minion was installed by
sudo ./bootstrap-salt.sh -A <master-ip-address> -i $(hostname) stable 3006.3. no custom config on minion - master runs inside a container using image
saltstack/salt:3006.3. master configs:
nodegroups:
prod-early-adopter: L@minion-hostname-1
prod-general-population: L@minion-hostname-2
release: L@minion-hostname-3
custom: L@minion-hostname-4
file_roots:
base:
- <path/to/custom/state/file>
state file:
pull_state_job:
schedule.present:
- function: state.apply
- maxrunning: 1
- when: 8:00pm
deploy:
cmd.run:
- name: '<custom-command-here>'
- runas: ubuntu
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior (Include debug logs if possible and relevant)
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3006.3
Python Version:
Python: 3.10.4 (main, Apr 20 2022, 01:21:48) [GCC 10.3.1 20210424]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: Not Installed
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: alpine 3.14.6
locale: utf-8
machine: x86_64
release: 5.11.0-1022-aws
system: Linux
version: Alpine Linux 3.14.6
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 11
- Comments: 172 (77 by maintainers)
Commits related to this issue
- Add changelog for issue #65265 — committed to dwoz/salt by dwoz 2 years ago
- Add changelog for issue #65265 — committed to dwoz/salt by dwoz 2 years ago
- Add changelog for issue #65265 — committed to saltstack/salt by dwoz 2 years ago
Everyone,
At least one user who could not upgrade to 3007.x fixed their environment with this patch:
Can anyone in this thread that is willing try out 3007.1 with this patch and report your findings. It would be a huge help.
Same issue upgraded master to 3007.1 and minions dropping like flyes. The crazy part, when i run tcpdump on minion and run
salt 'minion' test.pingit doesn’t even show single packet on minion, the connection is totally fucked. In netstat on master there can be seenestablishedandclose_waitconnection to 4505 from minion.salt-call from minion works ok, but it bypass the connection between master and minion. Since it create his own, for that call.
With latest releases salt becoming more and more unusable in prod enviroment. 3007.1 - minions doesn’t work 3007.0 - windows fucked 3006.6 - grains don’t work 3005 - salt-ssh was semi-working
So is very unconfortable upgrading/downgrading, when you need specific feature to work.
PS: Downgrade to 3006.8 master with 3007.x minions is working ok. Master 3007.x is not working as expected.
This seems to still be an issue on 3006.7 when both minion and master are the same version
We’re seeing the same issue with 3007.1 on both the master and the minions.
Watching with tcpdump on the master we see a whole bunch of the salt minions attempting to talk to the master on port 4505. They send the SYN packet, but never get SYN-ACK back, even though port 4505 was open and other systems are connected to the port. At this point were about 1000 file descripters in use by the salt master, but many of the sockets were in CLOSED_WAIT state.
We have LimitNOFILE set to 100000 in the salt-master.service file which should allow 100000 open sockets – but it smells like LimitNOFILE is not being honored by ZMQ.
Digging in the sourece, I find: grep -r ZMQ_MAX_SOCKETS * lib/python3.10/site-packages/zmq/backend/cython/constant_enums.pxi: enum: ZMQ_MAX_SOCKETS lib/python3.10/site-packages/zmq/include/zmq.h:#define ZMQ_MAX_SOCKETS 2 lib/python3.10/site-packages/zmq/include/zmq.h:#define ZMQ_MAX_SOCKETS_DFLT 1023
Where does salt explicitly set the socket limit for ZMQ? I can’t find if/where it does that – i.e. my thought is it’s unlikely that salt is setting anything, and is ending up with 1023 for the ZMQ socket limit!
Perhaps related to this: https://github.com/zeromq/libzmq/issues/4280 but, again, I can’t find where ZMQ_MAX_SOCKETS is ever set in the salt code.
After restarting the salt-master service, we see about 100 connections to 4505 and the same to 4506, and things work normally. I expect things will break once the number of file descriptors goes above 1023 again.
I’ve been testing in one of my environments with ~512 minions and my experience is similar.
Yup. Since upgrading my two big experiences have been:
Master will simply start timing out. i.e. running something like
salt 'node1' test.pingon the master or evensalt-call -l info test.pingon the master will tell me the master is timing out. CPU load is typically high at this point–like a load average of 4-6. Even firewalling off the box so no minions can communicate with it doesn’t resolve the load.Within minutes of restarting the master and using an external tool to restart all the minion services around 20% of the minions stop responding Just restarted everything a few minutes ago and did a
salt '*' test.ping. 507 minions targeted, 418 responded, 89 give the same error:Minion did not return. [No response]I thought the
tcp_keepalive_idleminion setting might fix it. I changed it from the default of 300 down to 60. Huge mistake. The moment that setting hit all the minions, the CPU load on my master (8 cores and 16 GB RAM) spiked up to ~15 and there was zero communication with any minion–not even locally using salt-call. I reverted the setting and tried to runsalt '*' state.sls salt.minionand nothing responded. I tried applying it locally viasalt-call -l info state.sls salt.minionand it just sat there hanged trying to talk to the master. A minute later the master itself started timing out. Restarts didn’t fix it. I ended up blocking the two port used by Salt in the firewall, and immediately the local salt-call command start running great and CPU load came down. I had to adjust the firewall for an hour slowing letting in new IPs and runningsalt '*' state.sls salt.minionto get them to stop trampling the master.I don’t think having two masters that “just can’t even” will help. 😉
I feel you. We’ve invested a ton of time over the past ~10 years automating systems using Saltstack. I love using it. …but over the lifetime of the 3006.x releases until now, we’ve had one disastrous release after another. There’s always something critical or important broken in Salt that causes problems in our environment…and honestly, it’s usually something related to the transport. ZMQ seems to be the wrong choice of transports in Salt.
I’d start testing and switch to websockets in a heartbeat…but at the moment it looks like you can only have one transport active on your master. So you either switch everything all at once, or you don’t. And unfortunately the installers don’t really support using different transports. i.e.
msiexec /i salt-package-for-windows.msi MASTER=salt.example.tlduses ZMQ. You don’t really have an option unless you want to somehow deploy the config to the box ahead of time and use the CUSTOM_CONFIG option to point to that config file or you “bootstrap it” viasalt-call -l info --master salt.example.tld state.sls salt.minionusing the salt formula.Not being an expert on salt transports…I with multi-transport was an option, or maybe an option to ditch zeromq and use a more reliably implemented queuing system like rabbitmq.
I had a recent very frustrating experience with lack of stability in the 3007.0 master. I’ve been attempting to upgrade one environment from 3001 to 3007 with about 1250 minions all running 3000. I was unable to to get a master to run stably on 3007.0. We had to fall back to the 3006.8 master to complete the upgrade.
Our Salt setup in this particular environment is very minimal: we’re only using Salt for its remote execution facilities - we aren’t running states at all. (In fact, what we do have defined for states and pillar are likely broken; we see errors about them in the logs, but it’s irrelevant to how we use Salt in this case.)
Our environment is based on RHEL 7 under VMware ESX VMs. We run the master in a container. Initially we attempted to replace the 3001 master with the 3007.0 master, but every time we brought up the master, it proved unable to talk to the minions. We saw the key requests come in and get auto approved, and as others have said, we were able to execute commands on the minion VMs via
salt-call, but pings or any other commands from the master failed, and we had to bring the 3007 master back up.After several attempts at upgrading the master whole cloth, we instead embarked on a strategy of setting up a separate VM with the 3007.0 master, and then using the 3001 master to update the minion config to attach to both masters. We started with small batches, first successfully attaching minions to both. We then attempted to use the 3007.0 master via
cmd.run_bgto update the minion config to only talk to 3007, and then to use yum upgrade to update the minions to 3007.0 as well. I tried using the-band--batch-waitparameters to space out the upgrades, but kept running into points where the batch would fail because the 3007 master was completely unable to talk to any of the minions. This happened with as few as 300-400 minions attached to the 3007 master. We found that somewhere above 900 minions we were unable to get the master to start successfully at all.Though minions checked in, the master was unable to communicate with them. For a time it seemed that stopping the container and completely removing the
/var/cache/saltdirectory helped, but the relief was brief: as soon as we started doing anything with multiple minions, the communication blackout would fall.We learned that restarting the master would allow connection to the minions for perhaps an hour, but then the master would lapse into being unable to talk to any minions. Minions could still communicate with masters, and from their point of view, they never lost the connection. For that matter, there was nothing in the master logs about any issues, either - just a failure to communicate.
Even though we’d never had any issues with our 3001 master and minion configs, I went through the advice about scaling Salt and made adjustments to the master VMs resources, taking it from 6GB ram and 4 processors to 8gb/8 processors. I also tried various combinations of tuning parameters in both the minion and master configs, but in the end though I was able to bring up the 3007.0 master with as many as 960 minions, the master was not stable - within an hour of starting, it would lose connection and stay that way for hours or days, and then mysteriously reconnect. After increasing the VM resources on the 3007 master, I never saw any evidence that the VM itself was under any serious load.
After 6 separate attempts over dozens of hours to get my 1250 minions upgraded, switching to a 3006.8 master is what did the trick. As soon as I brought up the 3006.8 master in place of my 3007.0 master, it was able to communicate with the already updated 3007.0 minions, and I was able to move the last few hundred VMs off my 3001 master and upgrade them to 3007.0 without any issues. That was my UAT environment, but now at least I have confidence that I should be able to complete my ~1100 minion production environment in a far more expeditious way.
The whole experience was rather sobering. In the next four months I’m going to have to update a far older version of Salt in four different environments. Those environments rely completely on Salt for all configuration from kickstarting to Kubernetes configuration. I think the dual master strategy could be helpful, but at this point, I’m definitely going to target 3006.8 rather than 3007 unless and until I hear these issues have been fixed.
@dnessett I found cause for this issue in my setup - lock inside
AsyncReqMessageClient._send_recvinMultiMinionProcessManager MinionProcessManagerprocess.In my case when minion tried to send data to master, minion called
_send_recvand hanged because it cannot acquire lock. While inspectingself.socketI noticed, that there is future insideself.socket._recv_futures. Looks like minion is awaiting data from master, but master for some reason did not receive an answer, making minion wait forever.self.socket.sendand waits until ZMQ sends messageself.socket.recvPotential solution is to add timeout to
self.socket.recv. I tried to add it myself, but minion is now raising errorZMQError('Operation cannot be accomplished in current state'). Probably this is the reason for this behavior.Patch:
Function
_send_recvdid not change in salt3007.0,3007.1, which may explain why @darkpixel experiencing this bug on3007.1. Probably we experiencing same bug, but with different conditions needed to reproduce.@q1x Given the current disarray of the project, I wouldn’t hold out too much hope. 😦
Hi @dwoz any indication of a date for a fix for this?
This patch worked great on a salt master on 3007.1 with a few thousand minions 👍
I’d say that fixes it @dwoz. All my minions are still online.
You have a fair point. I was thinking about the recently introduced master cluster concept. The write-up is brief, at least some of the information currently in
/var/cache/salt(at least on RHEL systems) that the master keeps is apparently moved into a shared file system. Presumably at least the load back from minions would be distributed, but given that the bug for me is that the master loses touch with the minions while the minions can still communicate with the master, so I suspect you’re right: this probably won’t help.It’s also a lot more complicated. I saw a table showing that at least in theory you could handle up to 8,000 minions on a master if you had more resource. We’re both far from that number.
I’m going to attempt to upgrade my production saltmaster and ~1,000 minions early tomorrow morning using the 3006.8 master. I’ll report back on how it went.
I can confirm! 3007.0 for salt master was for me aswell a total mess! On minion side the version is working fine! I downgraded my salt master also to 3006.8 and everything is working fine now… Hopefully a stable release for 3007 will soon arrive!
I downgraded the master and the minion running on the master to 3006.8 and semi-reliable connectivity appears to have been restored. All the minions are still running 3007.1 and appear to be working fine.
con_cachewas probably a red herring as it keeps dropping back to 0 and counting back up constantly.I didn’t use those exact settings because my master is smaller and has fewer minions. 8 cores * 1.5 threads per core = 12 threads = 2,400 minions (I only have ~700 on this test box).
It’s no longer dropping all the minions every few hours…it’s more like once or twice a week.
I seem to have a very similar issue with 3006.x but in my case restarting the master does not have any effect and only a minion restart resolves the issue.
Another oddity is that I can see in the minion logs that the minion is still receiving commands from the master and is able to execute them just fine but the master seemingly never receives the response data. If I issue a
salt '*' service.restart salt-minionfrom the master all of the minions receive the command and restart and pop back up just fine and then communication will work for probably another 12 hours or so.I don’t recall having this issue on 3005.x but I have not downgraded that far yet… so far both 3006.5 and 3006.4 minions have the problem for me. I’ll try to run a tcpdump if I have time
@darkpixel : Not every one can use 3007.x as that is LTS, so hoping for similar stable version for 3006.x too.
Following up on my previous comment, I made workaround around
ZMQError- recreate socket on error: (3006.x)Without recreation, socket is stuck in
PollEvent.POLLINstate (self.socket._state), that probably indicates that socket is still waiting for events, but is unable to send them (socket.send will raiseZMQError).With this patch, bug does not reproduce in my environment.
@bencarleyfs I am testing on
3006.10version because i cannot reproduce it on3007.x. I did not notice thatAsyncReqMessageClientwas deprecated in3007, thank you. After looking at3007source, i did not found usages of AsyncReqMessageClient.@darkpixel My patch is not complete, i added it as code that i tried to add and got
ZMQError('Operation cannot be accomplished in current state'). Patch will not fix issue, but will spam error if you got same bug as me.I just applied it to the master a few minutes ago. I have a few changes I’m going to be pushing to ~400 minions in about an hour. I’ll see if I can get it to fall over or not and let you know.
I decided to try the TCP keepalive settings:
and turn off the job that performs
test.pingfrom the master. The keepalive settings seem to work to keep my minions responding after several days, so I’ve disabled the script.FWIW, I have neither latency nor loss, all my minions are local, and still experience the issue.
I’m interested in what @gou177 said, that feels much closer to where the problem lies - i…e. that something, certain actions, certain data being sent, certain conditions, on the minion or perhaps master, is the cause.
I think whatever is being solved by the
tcp_keepaliveis a separate issue to the one being dealt with here.@Rosamaha1 tcp_keepalive: True Follow the above link In my scenario it worked try it generally this section is commented out in salt minion config /etc/salt/minion
The intermittent connection between the master and minion is mainly due to the response not coming from minion to master
When we execute a command in salt master to perform something in minion , the action is performed but the response is not coming to master thats the reason we get connection issue
Minion didnt return
Solution which worked for me
Keep this config in salt-minion
tcp_keepalive: True tcp_keepalive_idle: 60
These settings control TCP keepalive behavior, which helps maintain long-lived connections between the Salt Minion and the Salt Master
if there is no activity for 60 seconds, a keepalive packet will be sent to check if the connection is still alive
@darkpixel You may have wondered what happened to me and why I haven’t continued the discussion on this bug. I have been attempting to get the minionswarm test script working (the current version is very old and no longer works). See Minionswarm fails. Once I get it working properly, my goal is to attempt to use it to reliably reproduce this bug, so some progress is possible fixing it.
However, I did learn one thing while working on minionswarm that is relevant to this discussion. Master failover only works with zeromq transport. It does not work with tcp transport. So, if you switch to websocket, you will not be able to use master failover.
@darkpixel: I just read through all the comments from Sept 21, 2023 through the latest. I missed quite a few on my first read through, since a great number were hidden, which required me to click on a reveal button. I have a few questions. On Sept 24, 2024, dwoz provided a patch if await self._monitor_socket.poll(), that initially excited everyone. However, it appears it did not solve the problems you are experiencing. Can you give a quick explanation what happened when you tried this patch? Second, many of the reports involved configurations with hundreds of minions. It is infeasible to set up a test environment with hundreds of minions in order to analyze the bug. Can you suggest how to reproduce the bug with, say, one master and one minion? Or do you think the bug is related to a high number of minions per master?
After making a bunch of architectural changes, I found one way of hosing Salt.
If you use an external pillar provider (i.e. curl some JSON) and you either set it to delay responses by ~5,000 msec or return bad data (404, garbage HTML instead of JSON, TCP connection reset, etc…) or a combination of both, you can get the master and minions into a hanged state where the only way to bring them back (even after removing the delays and bad data) is to restart them.
You’ll get authentication timeouts, file client timeouts, event bus timeouts, etc… Not sure if it’s ZMQ that doesn’t any sort of delay or hiccup in network traffic or what…but it devastates my infrastructure if I induce it.
We will get things back on track. We were ready to do a release at the end of October but our CI/CD infrastructure went away with some service consolidation. We are currently working to get CI/CD pipelines going again and this effort is coming along. Hopefully it won’t be too much longer before we can start cutting releases again. Likely few weeks more before we are there.
Just applying it to the master is what worked for us
I completed my production upgrade of about 1,000 VMs from salt-3000 to 3007 without incident. As noted, however, I’m not running states on these VMs - just getting salt-minions to connect so I can run commands against them.
I did use the strategy of using the older saltmaster to do all the jobs in roughly three steps:
salt-cpover new minion configs and yum repo pointing to salt3007cmd.run_bgcopy new minion config pointing to old master and new master into /etc/salt/minion then runsalt-call service.restart salt-minionThis left me attached to both the minions.cmd.run_bgset new /etc/salt/minion config pointing to only new master, copy new yum repo into /etc/yum.repo.d pointing to salt-3007.0 minions, and runyum upgrade -y salt-minion. This upgraded the minion and restarted it pointing to my new 3006.8 master.I ran these in larger batches of about 250-300 minions, and then I ran the
cmd.run_bgcommands with batches of 20-25 with ``–batch-wait=15` to space them out.Load average on my 3006.8 master never went above 2.5 on my 8 cpu VM.
We’ll see what the future holds when I start upgrading my other salt environments where I actually depend on salt. Right now I’m going to stick to the 3006.8 master, but I’ll be closely watching 3007 release notes to see if they claim to have fixed anything that sounds like this issue.
Well shoot. More inclusive experimenting. I finally said “screw it–we’ll see if this is a resource issue”…and I spun up a VM with 32 cores and 256 GB of memory.
I then CSSH’d out to 56 minions and ran
salt-call -l debug state.highstate. Instead of 100% of them failing to authenticate to the master due to auth timeouts…only ~30% of them failed. The rest chugged away applying states…right up until 8 hanged on the x509 state. After 15 minutes they hadn’t budged, so I restarted the salt-minion in charge of handling certs. 5 minutes later still nothing. So I CTRL+C’d the hanged minions and tried again. They still hanged. Funny thing is, all 56 minions needed certs renewed, it’s just those 5 that didn’t.Then I decided to target ~500 minions. They all responded to pings. Then I tried a
saltutil.refresh_pillar. That worked too. Then I tried asaltutil.sync_alland the master timed out. The CPU load was 0.1 on the master the entire time.And the minions frequently say:
or
I tried a few more times and it kept timing out. I restarted it and it started working again.
Not wanting to spend $3.50/hr on a mostly working salt master, I scaled it back down to 16 GB RAM and 8 cores.
So it could be resources in some cases, but it still looks like there’s a fundamental connectivity problem.
Plus no one wants to spend $3k/mo on a virtual machine for a salt master than manages ~500 minions. 😉
Hmm…I noticed something interesting and potentially significant. I saw the
con_cachesetting in the config file that defaults to false. I figured I would turn it on and see what happened.After restarting the master, I get lots of entries like this in the log:
It sits there and counts up (if I’m idle or issuing a command like test.ping) until it hits about ~300-400 entries in the cache…then with no warnings or errors in the log it resets and starts counting up again in the middle of the flood of “Authentication requested from” and “Authentication accepted from” messages.
3007.1 is completely dead for me. Under 3006.7 minions would slowly become unavailable over a day or two until I ran a
minion.restart. Even though all commands to the minions returned "Minion did not return. [Not connected]``` they actually were connected and restarted themselves and started communicating properly.Now under 3007.1 (skipped 3007.0 because it was pretty well b0rked for Windows minions), minions disconnect after a few minutes.
If I restart the salt master and issue a command, I’m good. If I restart the salt master and wait ~5 minutes, all the minions are offline and won’t come back with a
minion.restart, only restarting the salt master.The salt master logs show a non-stop stream of “Authentication requested from” and “Authentication accepted from” messages. Typically I would get those messages right after restarting the 3006.7 master or after issuing a command like
salt '*' test.ping, but they’d settle down when nothing was going on.Now I’m getting 10-15 per second non-stop.
Using the minion on the master, I can view the logs and verify the minion doesn’t receive the
minion.restartcommand–but there are also no errors about communication issues with the master.Even stranger, I can connect out to a minion and manually run
state.highstateand it works perfectly fine. No issues communicating with the master there…just receiving commands I guess.I tried those settings @gregorg. It’s been intermittent for the last three days…and this morning 100% of my minions are offline (even the local one on the salt master)
If I connect to a box with a minion I show the service is running. I can totally state.highstate and everything works properly.
Restarting the master brings everything online.
There’s nothing that appears unusual in the master log. I can even see minions reporting their results if I do something like
salt '*' test.ping, but all I get back isMinion did not return. [Not connected].I’d love to switch to a potentially more reliable transport, but it looks like Salt can only have one transport active at a time…so if I enable something like websockets it looks like all my minions will be knocked offline until I reconfigure them.
It just happened on my master which is 3007.0…
I was going highstate on a minion that involves certificate signing and it refused to generate the certificate with no error messages in the salt master log.
I tried restarting the salt master, no dice.
About 10 minutes later I decided to restart the salt master’s minion…and suddenly certificate signing worked.
The minion on the master wasn’t communicating with the master…locally…on the same box…
3007.0 is…worse?
Woke up to all ~600 minions in an environment being offline.
salt '*' test.pingfailed for every minion.The log showed returns from every minion, but the master spit out
Minion did not return: [Not connected]for every single one.Restarted the salt-master service, got distracted for ~15 minutes, ran another
test.pingand several hundred failed to respond.Used Cluster SSH to connect in to every machine I can reach across the internet and restarted the salt-minion service and I’m down to a mix of ~60 (Windows, Linux, and BSD) that don’t respond and I can’t reach. Maybe 10 of them are 3006.7.
I’d love to test/switch to a different transport like websockets that would probably be more stable, but it appears to be “all or nothing”. If I switch to websockets on the master, it looks like every minion will disconnect unless I also update them to use websockets…and if I update them to use websockets and something breaks, I’m going to have to spend the next month trying to get access to hosts to fix salt-minion.
@darkpixel Yes I have found the same thing and have the same workflow. Something just gets stuck and responses get lost somewhere. They are always receiving events however as you say, in my experience
I just discovered something.
At any random time I might have 25-50 minions that don’t appear to respond to jobs. They may or may not respond to
test.ping, but definitely won’t respond to something likestate.sls somestate.…buuuut they ARE actually are listening to the master.
So my workflow is stupidly:
Restarting the saltmaster seems to fix the issue for all minions, for a while, but the issue will return after about 12 hours on a different seemingly random selection of minions.
It seems worse with 3006.5 with Linux as the master when managing Windows minions.