scipy: BUG: filter out unhelpful & noisy numpy warnings in implementation, not in tests
This was surfaced by #19804, causing me to take a look where those warnings were coming from. I was surprised that - in the example below at least - the warnings are duplicated between scipy & numpy (not caught by the new pytest 8 because the messages are identical).
Looking closer, we’re filtering out a lot of warnings in tests that should IMO really be filtered out in the implementations, rather than the tests.
As an example:
>>> from scipy import stats
>>> stats.skew([])
[...]/scipy/stats/_stats_py.py:1175: RuntimeWarning: Mean of empty slice.
mean = a.mean(axis, keepdims=True)
[...]/numpy/core/_methods.py:121: RuntimeWarning: invalid value encountered in divide
ret = um.true_divide(
[...]/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
[...]/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
ret = ret.dtype.type(ret / rcount)
nan
4 warnings here are at least 3 too many, and we need to filter at least the ones from numpy; though IMO even “Mean of empty slice” is not helpful, and we could perhaps just return np.nan without a warning.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 21 (21 by maintainers)
Commits related to this issue
- TST: account for parasitic warnings for compatibility with pytest 8 By and large, these should all be filtered out in the implementation, not in the tests, see GH-19805. — committed to h-vetinari/scipy by h-vetinari 3 years ago
- TST: account for parasitic warnings for compatibility with pytest 8 By and large, these should all be filtered out in the implementation, not in the tests, see GH-19805. — committed to h-vetinari/scipy by h-vetinari 3 years ago
- TST: account for parasitic warnings for compatibility with pytest 8 By and large, these should all be filtered out in the implementation, not in the tests, see #19805. — committed to h-vetinari/scipy by h-vetinari 3 years ago
- TST: account for parasitic warnings for compatibility with pytest 8 By and large, these should all be filtered out in the implementation, not in the tests, see #19805. — committed to h-vetinari/scipy by h-vetinari 3 years ago
- TST: account for parasitic warnings for compatibility with pytest 8 By and large, these should all be filtered out in the implementation, not in the tests, see #19805. — committed to h-vetinari/scipy by h-vetinari 3 years ago
- TST: account for parasitic warnings for compatibility with pytest 8 By and large, these should all be filtered out in the implementation, not in the tests, see #19805. — committed to tylerjereddy/scipy by h-vetinari 3 years ago
To be honest I don’t know. I just noted the output from pytest and copied it here. I’d be as surprised as you though if pytest shows warnings here that had explicitly been captured.
One thing to maybe check is your pytest version - v8 got much stricter if a function raises several warnings (rather than just taking the first one)
Yes, exactly. We didn’t think it made sense for a single short slice to cause the whole operation to error out.
In some contexts, yes. Then again, there’s also an argument to be made for consistency within SciPy and between SciPy and NumPy.
np.meanandnp.varcould have errored out, but for better or for worse, they set an example of returningnanwith empty input. That’s an appropriate result (if one is going to be returned) because the statistic is undefined. Many important SciPy functions (the moment functions, the t-tests, wilcoxon) followed suit. So it would make some sense for the rest to do the same, even in the 1D case, rather than the user having to guess.Cases in which there are enough obvservations to calculate a numerical value of a statistic but not enough for the traditional, asymptotic p-value approximation to be accurate are bit more gray. But in those cases, I’d say we should just give a permutation or Monte Carlo p-value[^1]. At this point, it’s very easy and quite fast to do so (e.g. see
methodofpearsonr).[^1]: The downsides are that the Monte Carlo p-values are stochastic and the permutation p-values, while they can be be deterministic and exact for small samples, technically change the interpretation of the test… There are solutions, but this is already getting too long : )
Having fiddled a little bit with special test suite recently, it is like a boeing 747. Wonderful to look at go, but almost impossible to do surgery on it or modernize. I still don’t know what it does in certain places and hoping the code is correct enough. So not sure what the right way to go about this since there are also warnings coming up from the internals of the tests.