scipy: BUG: Erratic results from RectBivariateSpline when smoothing large arrays

Describe your issue.

According to the documentation, RectBivariateSpline can be used for smoothing by setting s to a non zero value. This works on small grids, but on larger grids, using any s value other than zero produces a spline that evaluates to NaN everywhere. To be clear, this occurs on data that do not contain any NaN. The problem occurs without any warning or error message. The problem occurs irregularly (running the code sometimes work, but most of the time it does not).

The problem occurs on grid with dtype int as well as float.

The problem occurs on Windows 10 and mac OS.

Since the problem does not alway occur, I suspect it may be due to a failure to allocate memory. If enough memory is available, everything works fine. However, it the memory allocation fails, RectBivariateSpline returns a nonsensical result without informing the user about the memory error.

Reproducing Code Example

import numpy as np
from scipy import interpolate

s = 1  # any value greater than zero causes the problem
nx, ny = 1000, 1700    # Set to larger values to reproduce the problem more consistently
img = np.random.randint(0, 100, size=(nx, ny))
x = np.arange(nx)
y = np.arange(ny)
spl = interpolate.RectBivariateSpline(x, y, img, s=s)
print(np.isnan(spl(x, y)).any())

Error message

No error message

SciPy/NumPy/Python version information

1.9.3 1.23.5 sys.version_info(major=3, minor=10, micro=8, releaselevel=‘final’, serial=0)

1.10.0 1.24.1 sys.version_info(major=3, minor=10, micro=8, releaselevel=‘final’, serial=0)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Makes sort of sense but that’s not the final answer still. The ValueError is reasonable: s is too small. However it’s a bit worrying that s=1 gives some answer and s=2 does not. Normally I’d expect that it errors our of s is too small, and returns some result for larger values of s. This behavior seems to hint that your fix still papers over the underlying issue.

The patch also looks a tad strange with wildly different code flow for nx == nmaxx and nx == nxe. Given that nmaxx and nxe are most likely equal to each other, it’s very much not clear what is the effect of the control flow change.

I still think it’s worth figuring out what actually triggers the problem.

There was no recent changes in or around RectBivariateSpline. git blame concurs: https://github.com/scipy/scipy/blame/v1.14.1/scipy/interpolate/_fitpack2.py#L1526-L1611

I think this would be nice to add to some building from source docs. Not necessarily the Fortran info, may be similar info for more popular languages 😃.

This is in the plan already:)

However, I did not manage to add a command line switch to the build — IIRC it used to react to the FOPT env variable, but it does not seem to now. Will ping @rgommers for a build advice.

FFLAGS will work. Meson only supports bog-standard env vars and prefer command line switches for everything else. And it comes with docs even:) https://mesonbuild.com/Reference-tables.html#compiler-and-linker-flag-environment-variables

Note though that env vars only works from a clean build, because they affect the configure stage (i.e., meson setup) and not a partial rebuild. So you can either git clean -xdf and set the env vars you want, or alternatively you can edit the relevant meson.build entry and add fortran_args: -fbounds-check temporarily.

Note that -g is never needed, that’s already the default in dev.py and in general is controlled via setting the build type: https://mesonbuild.com/Builtin-options.html#details-for-buildtype.