spyder: Memory leak while having spyder run too much stuff or for too long.
Description
What steps will reproduce the problem?
Just having a couple of windows open, leave the program running for 1-2 days, sometimes hours and it will slow down real bad to a point pressing enter for new lines takes a couple of seconds. Obviously unacceptable, back to VScode until you guys fix this.
Versions
- Spyder version: 4.2.1
- Python version: 3.7.9
- Qt version: 5.12.10
- PyQt5 version: 5.12.3
- Operating System: Windows 10
Dependencies
# Mandatory:
atomicwrites >=1.2.0 : 1.4.0 (OK)
chardet >=2.0.0 : 4.0.0 (OK)
cloudpickle >=0.5.0 : 1.6.0 (OK)
diff_match_patch >=20181111 : 20200713 (OK)
intervaltree : None (OK)
IPython >=7.6.0 : 7.19.0 (OK)
jedi =0.17.2 : 0.17.2 (OK)
jsonschema >=3.2.0 : 3.2.0 (OK)
keyring >=17.0.0 : 21.5.0 (OK)
nbconvert >=4.0 : 6.0.7 (OK)
numpydoc >=0.6.0 : 1.1.0 (OK)
paramiko >=2.4.0 : 2.7.2 (OK)
parso =0.7.0 : 0.7.0 (OK)
pexpect >=4.4.0 : 4.8.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
psutil >=5.3 : 5.7.3 (OK)
pygments >=2.0 : 2.7.3 (OK)
pylint >=1.0 : 2.6.0 (OK)
pyls >=0.36.2;<1.0.0 : 0.36.2 (OK)
pyls_black >=0.4.6 : 0.4.6 (OK)
pyls_spyder >=0.3.0 : 0.3.0 (OK)
qdarkstyle >=2.8 : 2.8.1 (OK)
qtawesome >=0.5.7 : 1.0.2 (OK)
qtconsole >=5.0.1 : 5.0.1 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
rtree >=0.8.3 : 0.9.4 (OK)
setuptools >=39.0.0 : 51.0.0 (OK)
sphinx >=0.6.6 : 3.3.1 (OK)
spyder_kernels >=1.10.1;<1.11.0 : 1.10.1 (OK)
textdistance >=4.2.0 : 4.2.0 (OK)
three_merge >=0.1.1 : 0.1.1 (OK)
watchdog : 1.0.2 (OK)
zmq >=17 : 20.0.0 (OK)
# Optional:
cython >=0.21 : 0.29.21 (OK)
matplotlib >=2.0.0 : 3.3.3 (OK)
numpy >=1.7 : 1.19.3 (OK)
pandas >=1.1.1 : 1.1.5 (OK)
scipy >=0.17.0 : 1.5.4 (OK)
sympy >=0.7.3 : 1.7.1 (OK)
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 29 (14 by maintainers)
I got the very same problem.
@stevetracvc, that doesn’t seem related to Spyder. I mean, perhaps memory is not released with option 1. because the objects used to do the hyperparameter optimization are retained in memory after it finishes.
Could you try to run option 1. in a Jupyter notebook to check if the memory leak also shows up there? Since Spyder and Jupyter use the same architecture to run code, that would imply that your problem is not caused by Spyder.
@ccordoba12 the variable explorer didn’t seem to change anything. I played a bit with what peendebak said above, looking at all the objects gc can list. It seems like a huge part of the problem is the large amount of text that my script generates (I turned off the progress bars and that seems to have made a huge impact). However, doing something as simple as
does NOT replicate it, so it must be something more complicated.
Even after closing the consoles, something else still must be referencing these objects (spyder?). For example, there are still references to matplotlib pngs generated during my training:
Any thoughts on how I could release these references? After closing the console, or removing all plots, I’d expect them to be freed from memory. For example,
gc.get_referrers(gc.get_objects()[-95])(wheregc.get_objects()[-95]is a png from matplotlib) lists <spyder.plugins.plots.widgets.figurebrowser.FigureCanvas object at 0x7fe79495cdc0>Awesome thanks for the suggestion, I’ll let you know!