scipy: BUG: interpolate.griddata: memory leak in linear mode under Python 3.12

Describe your issue.

When running a particular test in our testsuite, specifically under python 3.12, its memory use balloons up to 16GB, while it barely needs 1~2GB under any older Python version. It’s sadly relatively inconclusive, but with all kinds of memory leak testers/profilers, everything points at the griddata interpolation function. Particularly this instance: https://github.com/OGGM/oggm/blob/7489be15c644b54c5f20501c38e654a47c63c53c/oggm/core/gis.py#L797

I was unable to produce any minimal example on how to reproduce this, it only happens in our entire framework of libraries, which makes it really hard to pinpoint down what is leaking where here.

I have already tried running this against the latest 1.14.x branch, since there was a memory leak fixed there recently. No change in memory balooning behaviour though compared to latest 1.13 version. I’ve also tried the binary wheels from pypi vs. our self-compiled version. Both leak in the same way.

I have packaged this all up into a docker image, which is available here, with instructions on how to trigger the issue: https://hub.docker.com/repository/docker/timoroth/oggm-latest-dev-debug/general The python version in there is built in debug mode, so pytest-leaks works. The likely culprit is linked above already.

One interesting and concerning thing I noticed when running this with pytest-leaks in action: At the end, it exits with the following assertion:

test_merge_simulated_thickness: leaked references: [233775], memory blocks: [136886], file descriptors: [7]
...../include/python3.12d/object.h:798: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7fc6e830f650 is freed>

Like stated before, this really only happens when running under python 3.12. The memory behaviour under python 3.11 or older is absolutely tame. If you want to verify, run the same reproducer from the Docker-Hub image in this docker image: ghcr.io/oggm/untested_base:py3.11

Reproducing Code Example

See above, sadly I've so far been unable to find any minimal reproducer for this.

Error message

None, if the machine has enough RAM, the test runs to completion successfully.
However, the memory is never released until the python process exits.
When testing for leaks with pytest-leaks, it exits with an assertion about a negative ref count at the very end, after completing the test runs.

SciPy/NumPy/Python version and system information

>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.14.0rc2.dev0+1360.67d743e 1.26.4 sys.version_info(major=3, minor=12, micro=3, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /usr/include/x86_64-linux-gnu/openblas-pthread/
    lib directory: /usr/lib/x86_64-linux-gnu/openblas-pthread/
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS=
      NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64
    pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
    version: 0.3.20
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /usr/include/x86_64-linux-gnu/openblas-pthread/
    lib directory: /usr/lib/x86_64-linux-gnu/openblas-pthread/
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS=
      NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64
    pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
    version: 0.3.20
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.12.0
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 11.4.0
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 11.4.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.10
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 11.4.0
  pythran:
    include directory: ../../pip-build-env-t7ypm4mq/overlay/lib/python3.12/site-packages/pythran
    version: 0.16.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /usr/local/pyenv/versions/3.12-dev-debug/bin/python3.12
  version: '3.12'

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Yeah, some update of something must have fixed this. I still have an environment with the issue stashes away as docker image, so I could slowly travel through time with it, and see which update fixes it. But so far I didn’t have the motivation to do so. I’m just glad we’re not stuck on 3.11.

One thing to correct myself on:

/usr/local/pyenv/versions/3.12-dev-debug/include/python3.12d/object.h:798: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7f3d68875f80 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x00007f3e88b692a8)

This also happens without pytest-leaks, just purely by running the test to completion on a debug build of python 3.12.

When directly calling LinearNDInterpolator, not much changes:

   800    429.7 MiB      0.0 MiB           1           try:
   801    429.7 MiB      0.0 MiB           1               inputval = np.ravel(dem[pok])
   802   1101.8 MiB    672.1 MiB           1               ip = LinearNDInterpolator(points, inputval, fill_value=np.nan, rescale=False)
   803   1105.8 MiB      4.0 MiB           1               ipr = ip(inter)
   804   1105.8 MiB      0.0 MiB           1               dem[pnan] = ipr
   805                                                 except ValueError:
   806                                                     raise InvalidDEMError('DEM interpolation not possible.')
   807   1105.8 MiB      0.0 MiB           1           if gdir is not None:
   808                                                     log.info(gdir.rgi_id + ': DEM needed interpolation.')
   809                                                     gdir.add_to_diagnostics('dem_needed_interpolation', True)
   810                                                     gdir.add_to_diagnostics('dem_invalid_perc',
   811                                                                             len(pnan[0]) / (nx * ny))
   812                                                 else:
   813   1105.8 MiB      0.0 MiB           1               diagnostics_dict['dem_needed_interpolation'] = True
   814   1105.8 MiB      0.0 MiB           1               diagnostics_dict['dem_invalid_perc'] = len(pnan[0]) / (nx * ny)
   815
   816   1105.8 MiB      0.0 MiB           1       isfinite = np.isfinite(dem)

This is the same log with scipy downgraded to the 1.13.1 binary wheel from pypi:

   800    517.7 MiB      0.0 MiB           1           try:
   801    517.7 MiB      0.0 MiB           1               inputval = np.ravel(dem[pok])
   802   1187.1 MiB    669.4 MiB           1               ip = LinearNDInterpolator(points, inputval, fill_value=np.nan, rescale=False)
   803   1187.1 MiB      0.0 MiB           1               ipr = ip(inter)
   804   1187.1 MiB      0.0 MiB           1               dem[pnan] = ipr
   805                                                 except ValueError:
   806                                                     raise InvalidDEMError('DEM interpolation not possible.')
   807   1187.1 MiB      0.0 MiB           1           if gdir is not None:
   808                                                     log.info(gdir.rgi_id + ': DEM needed interpolation.')
   809                                                     gdir.add_to_diagnostics('dem_needed_interpolation', True)
   810                                                     gdir.add_to_diagnostics('dem_invalid_perc',
   811                                                                             len(pnan[0]) / (nx * ny))
   812                                                 else:
   813   1187.1 MiB      0.0 MiB           1               diagnostics_dict['dem_needed_interpolation'] = True
   814   1187.1 MiB      0.0 MiB           1               diagnostics_dict['dem_invalid_perc'] = len(pnan[0]) / (nx * ny)
   815
   816   1187.1 MiB      0.0 MiB           1       isfinite = np.isfinite(dem)

I also tried downgrading to 1.12.0 this time, and the results are practically identical to the 1.13.1 results.

With a del thrown in there, nothing changes either:

   800    581.8 MiB      0.0 MiB           1           try:
   801    581.8 MiB      0.0 MiB           1               inputval = np.ravel(dem[pok])
   802   1188.2 MiB    606.4 MiB           1               ip = LinearNDInterpolator(points, inputval, fill_value=np.nan, rescale=False)
   803   1188.2 MiB      0.0 MiB           1               ipr = ip(inter)
   804   1188.2 MiB      0.0 MiB           1               del ip
   805   1188.2 MiB      0.0 MiB           1               dem[pnan] = ipr
   806                                                 except ValueError:
   807                                                     raise InvalidDEMError('DEM interpolation not possible.')
   808   1188.2 MiB      0.0 MiB           1           if gdir is not None:
   809                                                     log.info(gdir.rgi_id + ': DEM needed interpolation.')
   810                                                     gdir.add_to_diagnostics('dem_needed_interpolation', True)
   811                                                     gdir.add_to_diagnostics('dem_invalid_perc',
   812                                                                             len(pnan[0]) / (nx * ny))
   813                                                 else:
   814   1188.2 MiB      0.0 MiB           1               diagnostics_dict['dem_needed_interpolation'] = True
   815   1188.2 MiB      0.0 MiB           1               diagnostics_dict['dem_invalid_perc'] = len(pnan[0]) / (nx * ny)
   816
   817   1188.2 MiB      0.0 MiB           1       isfinite = np.isfinite(dem)

The more I look at this, the more it seems like the GC has been completely short-circuited or something. When I run the same kind of tests under python 3.11, there is the occasional line with a negative memory delta, indicating that the GC ran in the meantime. But on 3.12, there are only 0 or positive numbers, with the VERY rare small negative number, likely caused by native code doing something.