scipy: BUG: stats: underflow in geom distribution
Describe your issue.
stats.geom.rvs returns negative values when p is small. The support is strictly positive and should never return these values.
When validating this in R, I am able to get valid values for substantially smaller values of p and this should mirror that kind of performance.
Reproducing Code Example
from scipy.stats import geom
# This returns a valid output
geom.rvs(0.0000000001, random_state = 1234) #(2125986401)
# Adding an extra zero will cause it to fail as will any smaller number
geom.rvs(0.0000000001, random_state = 1234) #(-2147483648)
Error message
-2147483648
SciPy/NumPy/Python version and system information
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.10.1 1.23.5 sys.version_info(major=3, minor=10, micro=9, releaselevel='final', serial=0)
Build Dependencies:
blas:
detection method: pkgconfig
found: true
include directory: c:/opt/openblas/if_32/64/include
lib directory: c:/opt/openblas/if_32/64/lib
name: openblas
openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= PRESCOTT MAX_THREADS=4
pc file directory: c:/opt/openblas/if_32/64/lib/pkgconfig
version: 0.3.18
lapack:
detection method: pkgconfig
found: true
include directory: c:/opt/openblas/if_32/64/include
lib directory: c:/opt/openblas/if_32/64/lib
name: openblas
openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= PRESCOTT MAX_THREADS=4
pc file directory: c:/opt/openblas/if_32/64/lib/pkgconfig
version: 0.3.18
Compilers:
c:
commands: cc
linker: ld.bfd
name: gcc
version: 10.3.0
c++:
commands: c++
linker: ld.bfd
name: gcc
version: 10.3.0
cython:
commands: cython
linker: cython
name: cython
version: 0.29.33
fortran:
commands: gfortran
linker: ld.bfd
name: gcc
version: 10.3.0
pythran:
include directory: C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-v_zt9njv\overlay\Lib\site-packages\pythran
version: 0.12.1
Machine Information:
build:
cpu: x86_64
endian: little
family: x86_64
system: windows
cross-compiled: false
host:
cpu: x86_64
endian: little
family: x86_64
system: windows
Python Information:
path: C:\Users\runneradmin\AppData\Local\Temp\cibw-run-dskf9r9k\cp310-win_amd64\build\venv\Scripts\python.exe
version: '3.10'
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 21 (20 by maintainers)
I’m not necessarily saying that the warning message has to suggest any particular action, just that whether we issue a warning or not shouldn’t hinge on something that’s outside of a user’s control. We often don’t have nearly enough information about the user’s context to suggest the right place to intervene, which might be far upstream of our library function. People solve linear systems for lots of reasons in lots of different domains, and how the ill-conditioned matrix comes into play varies. The root cause might be because the mesh in the PDE discretization is less than ideal. It might be something that could be resolved by numerical preconditioning (possibly with a different kind of solver) regardless of where the matrix comes from.
No warning message has to be comprehensive, in fact, being short and internet-searchable can be more valuable because people can find documentation and discussions that help them explore their options with higher context.
I definitely agree that the saturation behavior should be documented. I still think it’s acceptable to do nothing except document the behavior (Matt’s findings concerning the parameter space is very informative to include, as well as an example of how one can check for saturation oneself).
But if we do want to have a warning, let me make one more pitch for doing it based on the passed input parameters rather than observing a saturated output. My main principle when it comes to warnings is that they need to be actionable for the user of the function. If the user can’t change anything under their control to avoid triggering the warning, don’t warn. As a kind of corollary, I think that when users make choices under their control that would lead to the warning-worthy outcomes depending on the things that are outside of their control, we should usually still warn them. Otherwise, we get heisenbugs that are hard to track down. A coder uses
geom.rvs()with these extreme parameters and it happens to work fine with the PRNG state they tested it, but then in actual use, there are a bunch of warnings. But only sometimes.There are two things that determine whether or not this
rvs()method saturates the output: the parameters and the PRNG state. Practically speaking, the user only controls the parameters, not the PRNG state (certainly not in a way to control whether or not it saturates, except in contrived, party-trick scenarios). Every PRNG state is equally valid (or invalid), so I think it behooves us to think in terms of the ensemble of possible results when deciding to warn, not just when a particular PRNG state happens to push the result to the saturation limit. Our finite arithmetic always distorts the probability distribution, even aside from saturation. Under this rubric, we consider it warning-worthy when the distortion is high enough (for some value of “enough”). Even when we have very reasonablepvalues like 0.3, we’re distorting the distribution. AFAICT, whenp==0.3, our algorithm can never generate the maximumint64integer (or even much smaller integers); it is not actually true that every parameter choice has a finite chance of saturating. We just don’t care about that particular distortion because the true probabilities were so low to begin with that the amount of distortion to the distribution is small. Working through the ziggurat constants, I think it’s really only possible to start saturating at aroundp=1e-18.Also, more specific to this case, when
p<eps, in addition to the saturation, we also start to observe wider and wider gaps between possible integer outputs below the saturation limit. That’s an additional distortion that is only possible to detect by looking at the input parameters and not the result.If I understand correctly, the proposed patch would change the results in this case from negative numbers, which are obviously not geometrically distributed, to positive numbers that are also not geometrically distributed, but less obviously so. If that’s correct, I’d prefer not to implement this. An explicit warning or error might be OK, but there would be a performance hit in the non-exceptional case. There is also the more general question of whether it would be preferable for discrete distributions to produce floats when integers would overflow. @tirthasheshpatel @chrisb83 - we would appreciate your opinions here.
Hi @eluerken-wm, thank you for reporting.
This is actually an error coming from NumPy. Behind the scene,
stats.geom.rvsis making a call to thegeometricdistribution from NumPy.Now, this issue is only present with the legacy random number generator interface. And if you use the new
np.random.Generator, the problem is fixed:And you can verify this in SciPy by passing a proper
Generatorobject, which is the recommended way:Hence, I am going to close this issue.