salt: [BUG] After Upgrading, Windows Minion RAM Usage 7GB+
Description
After upgrading to 3005 from 3002.9 we are seeing heavy Salt minion memory usage across all our Windows servers. On this server with enough memory (96GB), total RAM usage of the Salt Minion is around 7GB. On more memory constrained Windows instances, the Salt Minion eats nearly all the available memory and starts paging. For example, on a 4GB server, Salt RAM usage is around 3GB.
Get-Process | ? ProcessName -eq python| select @{Name="RAMUsage";Expression={$_.WorkingSet64 / 1MB}} | measure -Sum RAMUsage
Count : 55
Average :
Sum : 7331.484375
Maximum :
Minimum :
Property : RAMUsage
On this server there are 55 salt-minion processes, each using about 130MB of ram each:
Get-Process | ? ProcessName -eq python| select name, id, starttime, @{Name="RAMUsage";Expression={$_
.WorkingSet64 / 1MB}}
Name Id StartTime RAMUsage
---- -- --------- --------
python 1008 9/11/2022 2:49:49 PM 69.3203125
python 1784 9/12/2022 1:58:43 AM 136.7578125
python 2084 9/11/2022 2:20:25 PM 147.26953125
python 2396 9/13/2022 2:54:43 PM 137.4765625
python 3344 9/11/2022 2:40:27 PM 69.15625
python 3908 9/11/2022 7:44:28 PM 136.02734375
python 3980 9/12/2022 11:14:46 AM 136.01171875
python 4632 9/11/2022 2:07:02 PM 147.6015625
python 4964 9/12/2022 5:21:47 PM 136.1171875
python 5052 9/11/2022 1:36:55 PM 136.96875
python 6884 9/13/2022 5:56:07 PM 137.84375
python 7472 9/11/2022 2:30:02 PM 69.234375
python 7824 9/11/2022 1:36:21 PM 96.453125
python 8440 9/11/2022 10:49:07 PM 136.05078125
python 8500 9/12/2022 4:59:36 AM 133.9453125
python 9096 9/11/2022 2:54:43 PM 68.984375
python 10164 9/11/2022 2:44:14 PM 69.44921875
python 11224 9/11/2022 2:56:26 PM 135.015625
python 11256 9/11/2022 4:37:44 PM 135.6875
python 11328 9/14/2022 3:23:16 PM 138.13671875
python 11424 9/14/2022 6:10:12 AM 137.94921875
python 11588 9/12/2022 11:36:54 PM 134.8203125
python 11924 9/14/2022 9:11:45 AM 136.58984375
python 12084 9/12/2022 2:15:28 PM 136.20703125
python 12652 9/14/2022 3:09:44 AM 138.05859375
python 13232 9/12/2022 8:08:16 AM 135.95703125
python 13348 9/12/2022 8:30:22 PM 135.6484375
python 13364 9/13/2022 8:59:03 PM 137.234375
python 13444 9/13/2022 2:40:25 AM 136.33203125
python 13860 9/13/2022 5:47:04 AM 137.9609375
python 14012 9/13/2022 8:47:31 AM 137.5078125
python 14044 9/14/2022 12:05:50 AM 137.703125
python 14560 9/13/2022 11:53:10 AM 137.48046875
python 14832 9/14/2022 12:14:59 PM 137.90234375
python 14964 9/15/2022 9:47:06 AM 139.140625
python 15376 9/15/2022 3:40:28 AM 138.2890625
python 15592 9/15/2022 6:57:40 PM 142.01171875
python 16932 9/14/2022 6:27:59 PM 138.32421875
python 17332 9/14/2022 9:28:52 PM 138.66796875
python 17432 9/15/2022 6:46:26 AM 138.46484375
python 17440 9/15/2022 12:53:42 PM 139.921875
python 17464 9/15/2022 12:31:12 AM 136.81640625
python 17700 9/16/2022 10:25:45 AM 148.3984375
python 18484 9/15/2022 10:01:28 PM 141.93359375
python 18524 9/15/2022 3:55:10 PM 141.26953125
python 19492 9/16/2022 7:35:29 PM 151.53125
python 19716 9/16/2022 7:17:40 AM 145.18359375
python 19808 9/16/2022 1:03:17 AM 143.23828125
python 19904 9/16/2022 1:30:48 PM 148.578125
python 19916 9/16/2022 4:11:25 AM 143.73046875
python 20848 9/17/2022 4:52:24 AM 155.0625
python 20952 9/16/2022 4:31:28 PM 149.3046875
python 21808 9/17/2022 1:44:41 AM 152.70703125
python 22220 9/16/2022 10:41:58 PM 151.81640625
python 22304 9/17/2022 7:56:50 AM 156.234375
Get-Date
Saturday, September 17, 2022 10:18:59 AM
Setup
Windows 2016 instances (both bare metal and virtualized via Hyper-V). Version 3005, via the MSI installer.
Steps to Reproduce the behavior
Nothing interesting in the logs.
Minion configuration is close to defaults:
schedule:
__mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,
minutes: 60, return_job: false, run_on_start: true}
grains_cache: true
# Not documented.
# https://github.com/saltstack/salt/issues/48773
disable_grains:
- esxi
disable_modules:
- vsphere
# Just annoying:
- boto3_elasticsearch
- victorops
- ifttt
- pushbullet
- zfs
Expected behavior
RAM usage should be constrained.
Screenshots
These servers are Core instances, so not much of a UI. This is a screenshot from a Hypervisor (96GB of ram, 7GB used by Salt).
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)C:\Program Files\Salt Project\Salt\bin\lib\site-packages\_distutils_hack\__init__.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
Salt Version:
Salt: 3005
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.7
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: Not Installed
Mako: 1.1.4
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
Python: 3.8.13 (tags/v3.8.13:ea67321, Aug 22 2022, 17:16:10) [MSC v.1929 64 bit (AMD64)]
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 19.0.0
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist:
locale: cp1252
machine: AMD64
release: 2016Server
system: Windows
version: 2016Server 10.0.14393 SP0 Multiprocessor Free
Additional context
All Windows instances appear to be impacted.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
Does this help?
Highstate output for a Windows node (Hyper-V host)
I don’t think any processes are hanging, at least there’s no impact to any operations. The process details suggest it’s a scheduled highstate.
It appears this occurs just with Highstate, no other execution are normally occurring on these servers.
Aside, I am hoping these servers will be upgraded to Windows Server 2022 soon’ish. There’s already a migration to upgrade to Ubuntu 22.04 - while refactoring the existing Salt states (migrating to onedir and removing some legacy workarounds for random Salt problems over the may years).