salt: [BUG] Memory regression with 3002

Description After upgrading 3001 to 3002 massively increased runtime for highstates has been observed.

A 3001 minion highstate:

[root@bb2 ~]# time salt-call state.highstate
...
Summary for local
--------------
Succeeded: 259 (changed=4)
Failed:      0
--------------
Total states run:     259
Total run time:    90.437 s

real	2m22.719s
user	0m23.485s
sys	0m3.956s
[root@bb2 ~]#

The same host once the minion and the master have been upgraded to 3002:

[root@bb2 ~]# time salt-call state.highstate
...
Summary for local
Summary for local
--------------
Succeeded: 255 (changed=4)
Failed:      1
--------------
Total states run:     256
Total run time:   279.516 s

real	9m33.315s
user	0m22.670s
sys	0m3.968s
[root@bb2 ~]#

The failure was a service error:

----------
          ID: service_iptables
    Function: service.running
        Name: iptables
      Result: False
     Comment: Running scope as unit run-15222.scope.
              Job for iptables.service failed because the control process exited with error code. See "systemctl status iptables.service" and "journalctl -xe" for details.
     Started: 23:01:31.466069
    Duration: 97.186 ms
     Changes:

Possibly unrelated…

Setup

Salt Version:
           Salt: 3001.1

Dependency Versions:
           cffi: 1.13.0
       cherrypy: 5.6.0
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.1
        libgit2: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: 1.3.12
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Apr  2 2020, 13:34:55)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: centos 7 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.19.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7 Core

Master and Minion are using the same stack, when running on 3002 the libraries are unchanged.

Downgrading to 3001 resolves the issue.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (17 by maintainers)

Most upvoted comments

@OrangeDog @whytewolf @tuxthepenguin84 @Arderos @jbouter Have a look at https://github.com/ixs/salt-58791/tree/master/archive/fba062ce8e96d9fe657fdc7cc8ce808cc7a4aacd and feel free to try out the centos7 RPMS there or the salt tarball. That has regular memory consumption for me, I bet it also fixes your issues.

ixs on Oct 25, 2020

Reverting the following changes brings memory consumption back down to 90MB. GNU time says Maximum resident set size (kbytes): 91688 which is fine.

5c15def711d1460703b951655dca9c3f0e0e4c10 (Test loader namespaceing)
26b5714f57eaae5d4895d3d03c233c350f241c90 (Honor loaded_base_name)
a6921d6a56110eb5d003b141354998eb07fb3c4f (Clean up sys.modules when the loader is GC’ed)

I’d recommend rolling a 3002.1 release ASAP.

RPMS and a python sdist tarball and logs is at https://github.com/ixs/salt-58791/tree/master/archive/fba062ce8e96d9fe657fdc7cc8ce808cc7a4aacd

ixs on Oct 25, 2020

@cachedout as requested on slack…

ixs on Oct 22, 2020