salt: Memory leak in salt-master

Description of Issue/Question

I have noticed a substianaital memory leak in salt-master

root@salt:~# ps axuf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       896  0.0  1.6 170836 33792 ?        Ss   Mar17   0:00 /usr/bin/python /usr/bin/salt-master
root      1240  0.0  1.4 246328 29472 ?        Sl   Mar17   0:00  \_ /usr/bin/python /usr/bin/salt-master
root      1241  0.0  1.5 247080 31888 ?        Sl   Mar17   0:00  \_ /usr/bin/python /usr/bin/salt-master
root      1252  0.1  3.0 451776 61828 ?        Sl   Mar17   1:50  \_ /usr/bin/python /usr/bin/salt-master
root      1321  0.1  1.6 794880 34604 ?        Sl   Mar17   1:52  \_ /usr/bin/python /usr/bin/salt-master
root      1322  0.0  1.4 170844 30336 ?        S    Mar17   0:00  \_ /usr/bin/python /usr/bin/salt-master
root      1327  0.0  1.5 547700 32176 ?        Sl   Mar17   0:00      \_ /usr/bin/python /usr/bin/salt-master
root      1330  0.0 10.3 977768 211292 ?       Sl   Mar17   0:01      \_ /usr/bin/python /usr/bin/salt-master
root      1338  0.0 10.3 999916 211612 ?       Sl   Mar17   0:01      \_ /usr/bin/python /usr/bin/salt-master
root      1339  0.0 10.3 978200 211584 ?       Sl   Mar17   0:01      \_ /usr/bin/python /usr/bin/salt-master
root      1345  0.0 10.3 978056 212132 ?       Sl   Mar17   0:01      \_ /usr/bin/python /usr/bin/salt-master
root      1350  0.0 10.3 982184 211692 ?       Sl   Mar17   0:01      \_ /usr/bin/python /usr/bin/salt-master

It eventually OOMs and/or kills the box.

Setup

all states and pillars are in gitfs roots, otherwise pretty standard.

Steps to Reproduce Issue

Leave the master running for 24 hrs.

Versions Report

root@salt:~# salt --versions
Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.0.5
           cffi: 0.8.6
       cherrypy: 4.0.0
       dateutil: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-79-generic
         system: Ubuntu 14.04 trusty

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 1
Comments: 24 (22 by maintainers)

Most upvoted comments

We have a similar issue, the salt-master suddenly consuming 5GB of memory for a mostly idle system with only a handful of minions. We’re not using any of the git modules. We don’t have pillars nor states, it’s basically a default installation on C7 that we use to manually execute commands. One distinctive thing is that there are often new minions that keep trying to authenticate and never get their key accepted (those are test systems), but we’re talking about ~10 minions max, so is fairly low load.

# salt --versions
Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: 1.4.2
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-229.20.1.el7.x86_64
         system: CentOS Linux 7.2.1511 Core

I am considering trying to plug Heapy (http://guppy-pe.sourceforge.net/) to our salt master to be able to dump stats on memory on demand. Is there anything already in the source for memory profiling?

moises-silva on Mar 23, 2016