spyder: Memory leak while having spyder run too much stuff or for too long.

Description

What steps will reproduce the problem?

Just having a couple of windows open, leave the program running for 1-2 days, sometimes hours and it will slow down real bad to a point pressing enter for new lines takes a couple of seconds. Obviously unacceptable, back to VScode until you guys fix this.

Versions

  • Spyder version: 4.2.1
  • Python version: 3.7.9
  • Qt version: 5.12.10
  • PyQt5 version: 5.12.3
  • Operating System: Windows 10

Dependencies


# Mandatory:
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree                    :  None (OK)
IPython >=7.6.0                 :  7.19.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  21.5.0 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.7.3 (OK)
pygments >=2.0                  :  2.7.3 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.2 (OK)
qtconsole >=5.0.1               :  5.0.1 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  51.0.0 (OK)
sphinx >=0.6.6                  :  3.3.1 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.10.1 (OK)
textdistance >=4.2.0            :  4.2.0 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog                        :  1.0.2 (OK)
zmq >=17                        :  20.0.0 (OK)

# Optional:
cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.3 (OK)
numpy >=1.7                     :  1.19.3 (OK)
pandas >=1.1.1                  :  1.1.5 (OK)
scipy >=0.17.0                  :  1.5.4 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Comments: 29 (14 by maintainers)

Most upvoted comments

I got the very same problem.

I’m encountering a persistent memory problem, similar to one reported years ago in this issue, where the pylsp process progressively consumes all the RAM (64 GB) when using Spyder IDE, specifically in environments with fastai installed. This problem occurs on two separate machines, and does not affect other environments.

Symptoms:

  • The pylsp process is connected to127.0.0.x and consumes significant RAM, leading to system instability and Spyder crashes.
  • Leaving the pylsp process running slows down execution, functions lookup, and in general the whole IDE. Eventually, it will eat the whole RAM and lead to system or IDE sudden crash.
  • Terminating the pylsp process frees up RAM and temporarily resolves the issue but disables important features like linting and completion. Furthermore, Spyder restarts the process within seconds, reintroducing the problem.
  • Disabling LSP from Spyder’s settings circumvents the RAM issue but at the cost of disabling essential coding features.

This issue forces me to use Spyder solely for code editing and basic execution, while I resort to running scripts for model training and testing via the terminal to avoid crashes and data loss.

Temporary Workaround: Currently, my solution is to disable the LSP within Spyder settings. This action, while effective at preventing memory leaks, significantly limits my ability to use Spyder for a full development workflow, especially for features like variable and class definition lookups.

My environment has been exported with conda list --export > environment_packages.txt and can be found here. OS is Ubuntu 22.04.

Any idea if this is related to the issues mentioned above, or yet another (separate) memory leak issue with Spyder?

@stevetracvc, that doesn’t seem related to Spyder. I mean, perhaps memory is not released with option 1. because the objects used to do the hyperparameter optimization are retained in memory after it finishes.

Could you try to run option 1. in a Jupyter notebook to check if the memory leak also shows up there? Since Spyder and Jupyter use the same architecture to run code, that would imply that your problem is not caused by Spyder.

@ccordoba12 the variable explorer didn’t seem to change anything. I played a bit with what peendebak said above, looking at all the objects gc can list. It seems like a huge part of the problem is the large amount of text that my script generates (I turned off the progress bars and that seems to have made a huge impact). However, doing something as simple as

while True:
    print("all work and no play makes Jack a dull boy")

does NOT replicate it, so it must be something more complicated.

Even after closing the consoles, something else still must be referencing these objects (spyder?). For example, there are still references to matplotlib pngs generated during my training:

{'fig': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x06\xaa\x00\x00\x02\x99\x08\x06\x00\x00\x00\xc0\xb2z\xeb\x00\x00\x009tEXtSoftware\x00Matplotlib version3.5.1, https://matplotlib.org/\xd8a\xf2\xbd\x00\x00\x00\tpHYs\x00\x00\x0b\x13\x00\x00\x0b\x13\x01\x00\x9a\x9c\x18\x00\x00\xf1\xfeIDATx\x9c\xec\xddwX\x14\xd7\xfb>\xfe\x9b&\x02\x0b,M\x01A\xb0\xa3\xd8b\x8b\x1d5\xf6\xde+\x8a\x1a[\xd4h\x12\x8d\x1a5v\x8d\xfd\x9d\xa8I\x8cQ\xb1w\xa3\xc6\xae1\xd6X\xa2&j\xb0b\x01\x15\xb0\x00\x02\x02\xd2\x9f\xdf\x1f\xfe\x98/\x0b\xdb\xb0\xad\xc9\xe7~]\x17\x97\xee\x9c)g\xce\x9c9s\

...

'fmt': 'image/png', 'fwidth': 1706, 'fheight': 665, '_blink_flag': False, '_qpix_orig': <PyQt5.QtGui.QPixmap object at 0x7f2b6c0456d0>, '_qpix_scaled': <PyQt5.QtGui.QPixmap object at 0x7f2b6c063740>}

Any thoughts on how I could release these references? After closing the console, or removing all plots, I’d expect them to be freed from memory. For example, gc.get_referrers(gc.get_objects()[-95]) (where gc.get_objects()[-95] is a png from matplotlib) lists <spyder.plugins.plots.widgets.figurebrowser.FigureCanvas object at 0x7fe79495cdc0>

Awesome thanks for the suggestion, I’ll let you know!