scipy: BUG: interpolate.griddata: memory leak in linear mode under Python 3.12
Describe your issue.
When running a particular test in our testsuite, specifically under python 3.12, its memory use balloons up to 16GB, while it barely needs 1~2GB under any older Python version. It’s sadly relatively inconclusive, but with all kinds of memory leak testers/profilers, everything points at the griddata interpolation function. Particularly this instance: https://github.com/OGGM/oggm/blob/7489be15c644b54c5f20501c38e654a47c63c53c/oggm/core/gis.py#L797
I was unable to produce any minimal example on how to reproduce this, it only happens in our entire framework of libraries, which makes it really hard to pinpoint down what is leaking where here.
I have already tried running this against the latest 1.14.x branch, since there was a memory leak fixed there recently. No change in memory balooning behaviour though compared to latest 1.13 version. I’ve also tried the binary wheels from pypi vs. our self-compiled version. Both leak in the same way.
I have packaged this all up into a docker image, which is available here, with instructions on how to trigger the issue: https://hub.docker.com/repository/docker/timoroth/oggm-latest-dev-debug/general The python version in there is built in debug mode, so pytest-leaks works. The likely culprit is linked above already.
One interesting and concerning thing I noticed when running this with pytest-leaks in action: At the end, it exits with the following assertion:
test_merge_simulated_thickness: leaked references: [233775], memory blocks: [136886], file descriptors: [7]
...../include/python3.12d/object.h:798: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7fc6e830f650 is freed>
Like stated before, this really only happens when running under python 3.12. The memory behaviour under python 3.11 or older is absolutely tame.
If you want to verify, run the same reproducer from the Docker-Hub image in this docker image: ghcr.io/oggm/untested_base:py3.11
Reproducing Code Example
See above, sadly I've so far been unable to find any minimal reproducer for this.
Error message
None, if the machine has enough RAM, the test runs to completion successfully.
However, the memory is never released until the python process exits.
When testing for leaks with pytest-leaks, it exits with an assertion about a negative ref count at the very end, after completing the test runs.
SciPy/NumPy/Python version and system information
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.14.0rc2.dev0+1360.67d743e 1.26.4 sys.version_info(major=3, minor=12, micro=3, releaselevel='final', serial=0)
Build Dependencies:
blas:
detection method: pkgconfig
found: true
include directory: /usr/include/x86_64-linux-gnu/openblas-pthread/
lib directory: /usr/lib/x86_64-linux-gnu/openblas-pthread/
name: openblas
openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS=
NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64
pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
version: 0.3.20
lapack:
detection method: pkgconfig
found: true
include directory: /usr/include/x86_64-linux-gnu/openblas-pthread/
lib directory: /usr/lib/x86_64-linux-gnu/openblas-pthread/
name: openblas
openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS=
NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64
pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
version: 0.3.20
pybind11:
detection method: config-tool
include directory: unknown
name: pybind11
version: 2.12.0
Compilers:
c:
commands: cc
linker: ld.bfd
name: gcc
version: 11.4.0
c++:
commands: c++
linker: ld.bfd
name: gcc
version: 11.4.0
cython:
commands: cython
linker: cython
name: cython
version: 3.0.10
fortran:
commands: gfortran
linker: ld.bfd
name: gcc
version: 11.4.0
pythran:
include directory: ../../pip-build-env-t7ypm4mq/overlay/lib/python3.12/site-packages/pythran
version: 0.16.1
Machine Information:
build:
cpu: x86_64
endian: little
family: x86_64
system: linux
cross-compiled: false
host:
cpu: x86_64
endian: little
family: x86_64
system: linux
Python Information:
path: /usr/local/pyenv/versions/3.12-dev-debug/bin/python3.12
version: '3.12'
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (7 by maintainers)
Yeah, some update of something must have fixed this. I still have an environment with the issue stashes away as docker image, so I could slowly travel through time with it, and see which update fixes it. But so far I didn’t have the motivation to do so. I’m just glad we’re not stuck on 3.11.
One thing to correct myself on:
This also happens without pytest-leaks, just purely by running the test to completion on a debug build of python 3.12.
When directly calling LinearNDInterpolator, not much changes:
This is the same log with scipy downgraded to the 1.13.1 binary wheel from pypi:
I also tried downgrading to 1.12.0 this time, and the results are practically identical to the 1.13.1 results.
With a
delthrown in there, nothing changes either:The more I look at this, the more it seems like the GC has been completely short-circuited or something. When I run the same kind of tests under python 3.11, there is the occasional line with a negative memory delta, indicating that the GC ran in the meantime. But on 3.12, there are only 0 or positive numbers, with the VERY rare small negative number, likely caused by native code doing something.