spyder: Memory leak while having spyder run too much stuff or for too long.
Description
What steps will reproduce the problem?
Just having a couple of windows open, leave the program running for 1-2 days, sometimes hours and it will slow down real bad to a point pressing enter for new lines takes a couple of seconds. Obviously unacceptable, back to VScode until you guys fix this.
Versions
- Spyder version: 4.2.1
- Python version: 3.7.9
- Qt version: 5.12.10
- PyQt5 version: 5.12.3
- Operating System: Windows 10
Dependencies
# Mandatory:
atomicwrites >=1.2.0 : 1.4.0 (OK)
chardet >=2.0.0 : 4.0.0 (OK)
cloudpickle >=0.5.0 : 1.6.0 (OK)
diff_match_patch >=20181111 : 20200713 (OK)
intervaltree : None (OK)
IPython >=7.6.0 : 7.19.0 (OK)
jedi =0.17.2 : 0.17.2 (OK)
jsonschema >=3.2.0 : 3.2.0 (OK)
keyring >=17.0.0 : 21.5.0 (OK)
nbconvert >=4.0 : 6.0.7 (OK)
numpydoc >=0.6.0 : 1.1.0 (OK)
paramiko >=2.4.0 : 2.7.2 (OK)
parso =0.7.0 : 0.7.0 (OK)
pexpect >=4.4.0 : 4.8.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
psutil >=5.3 : 5.7.3 (OK)
pygments >=2.0 : 2.7.3 (OK)
pylint >=1.0 : 2.6.0 (OK)
pyls >=0.36.2;<1.0.0 : 0.36.2 (OK)
pyls_black >=0.4.6 : 0.4.6 (OK)
pyls_spyder >=0.3.0 : 0.3.0 (OK)
qdarkstyle >=2.8 : 2.8.1 (OK)
qtawesome >=0.5.7 : 1.0.2 (OK)
qtconsole >=5.0.1 : 5.0.1 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
rtree >=0.8.3 : 0.9.4 (OK)
setuptools >=39.0.0 : 51.0.0 (OK)
sphinx >=0.6.6 : 3.3.1 (OK)
spyder_kernels >=1.10.1;<1.11.0 : 1.10.1 (OK)
textdistance >=4.2.0 : 4.2.0 (OK)
three_merge >=0.1.1 : 0.1.1 (OK)
watchdog : 1.0.2 (OK)
zmq >=17 : 20.0.0 (OK)
# Optional:
cython >=0.21 : 0.29.21 (OK)
matplotlib >=2.0.0 : 3.3.3 (OK)
numpy >=1.7 : 1.19.3 (OK)
pandas >=1.1.1 : 1.1.5 (OK)
scipy >=0.17.0 : 1.5.4 (OK)
sympy >=0.7.3 : 1.7.1 (OK)
About this issue
- Original URL
- State: open
- Created 6 years ago
- Comments: 29 (14 by maintainers)
I got the very same problem.
I’m encountering a persistent memory problem, similar to one reported years ago in this issue, where the
pylspprocess progressively consumes all the RAM (64 GB) when using Spyder IDE, specifically in environments withfastaiinstalled. This problem occurs on two separate machines, and does not affect other environments.Symptoms:
pylspprocess is connected to127.0.0.x and consumes significant RAM, leading to system instability and Spyder crashes.pylspprocess running slows down execution, functions lookup, and in general the whole IDE. Eventually, it will eat the whole RAM and lead to system or IDE sudden crash.This issue forces me to use Spyder solely for code editing and basic execution, while I resort to running scripts for model training and testing via the terminal to avoid crashes and data loss.
Temporary Workaround: Currently, my solution is to disable the LSP within Spyder settings. This action, while effective at preventing memory leaks, significantly limits my ability to use Spyder for a full development workflow, especially for features like variable and class definition lookups.
My environment has been exported with
conda list --export > environment_packages.txtand can be found here. OS is Ubuntu 22.04.Any idea if this is related to the issues mentioned above, or yet another (separate) memory leak issue with Spyder?
@stevetracvc, that doesn’t seem related to Spyder. I mean, perhaps memory is not released with option 1. because the objects used to do the hyperparameter optimization are retained in memory after it finishes.
Could you try to run option 1. in a Jupyter notebook to check if the memory leak also shows up there? Since Spyder and Jupyter use the same architecture to run code, that would imply that your problem is not caused by Spyder.
@ccordoba12 the variable explorer didn’t seem to change anything. I played a bit with what peendebak said above, looking at all the objects gc can list. It seems like a huge part of the problem is the large amount of text that my script generates (I turned off the progress bars and that seems to have made a huge impact). However, doing something as simple as
does NOT replicate it, so it must be something more complicated.
Even after closing the consoles, something else still must be referencing these objects (spyder?). For example, there are still references to matplotlib pngs generated during my training:
Any thoughts on how I could release these references? After closing the console, or removing all plots, I’d expect them to be freed from memory. For example,
gc.get_referrers(gc.get_objects()[-95])(wheregc.get_objects()[-95]is a png from matplotlib) lists <spyder.plugins.plots.widgets.figurebrowser.FigureCanvas object at 0x7fe79495cdc0>Awesome thanks for the suggestion, I’ll let you know!