scipy: BUG: stats: distribution methods emit unnecessary warnings from Boost

Describe your issue

Several statistical distribution methods emit warnings stemming from Boost. This issue tracks the status of all of them.

beta.ppf as reported here; should be fixed by boostorg/math#827
ncf as reported in gh-17101, should be fixed by boostorg/math#846
nct - I’m not sure that this has been reported separately yet. Is it caused by the same sort of thing?
ncx2 - Ditto.

gh-17272 will silence the failures in SciPy’s tests temporarily, but as many of these marks as possible should be removed when gh-17207 merges.

Original Post

Describe your issue.

macOS CI is failing due to an overflow. I am still trying to reproduce locally.

NumPy was updated to 1.21.3, could be this or another dependency.

https://github.com/scipy/scipy/runs/3960228039

Reproducing Code Example

# from scipy/stats/tests/test_distributions.py:2878: in test_issue_12796
import numpy as np
from scipy import stats
q = 0.999995
a = np.array([ 2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
b = np.array([99999, 99998, 99997, 99996, 99995, 99994, 99993, 99992, 99991, 99990, 99989, 99988, 99987, 99986, 99985, 99984, 99983, 99982, 99981])
stats.beta.ppf(q, a, b)

Error message

FAILED scipy/stats/tests/test_distributions.py::TestBeta::test_issue_12635 - ...
927
FAILED scipy/stats/tests/test_distributions.py::TestBeta::test_issue_12794 - ...
928
FAILED scipy/stats/tests/test_distributions.py::TestBeta::test_issue_12796 - ...
...
scipy/stats/_continuous_distns.py:626: in _ppf
916
    return _boost._beta_ppf(q, a, b)
917
E   RuntimeWarning: overflow encountered in _beta_ppf

SciPy/NumPy/Python version information

master on macOS/CI

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 55 (52 by maintainers)

Commits related to this issue

TST: silence test failures on macOS for `beta.ppf` overflow See gh-14901 for details. — committed to rgommers/scipy by rgommers 3 years ago
TST: silence test failures on macOS for `beta.ppf` overflow See gh-14901 for details. [skip azp] [skip github] — committed to rgommers/scipy by rgommers 3 years ago
TST: silence test failures on macOS for `beta.ppf` overflow (#14938) * TST: silence test failures on macOS for `beta.ppf` overflow See gh-14901 for details. [skip azp] [skip github] Co-auth... — committed to scipy/scipy by rgommers 3 years ago
TST: silence test failures on macOS for `beta.ppf` overflow (#14938) * TST: silence test failures on macOS for `beta.ppf` overflow See gh-14901 for details. [skip azp] [skip github] Co-auth... — committed to tylerjereddy/scipy by rgommers 3 years ago
MAINT: filter RuntimeWarnings in stats functions Fixes test suite failures on macOS 12, arm64. See gh-14901 for details. The failures on x86-64 may not be identical, and this doesn't close the issue... — committed to rgommers/scipy by rgommers 3 years ago
MAINT: filter RuntimeWarnings in stats functions Fixes test suite failures on macOS 12, arm64. See gh-14901 for details. The failures on x86-64 may not be identical, and this doesn't close the issue... — committed to rgommers/scipy by rgommers 3 years ago
MAINT: filter RuntimeWarnings in stats functions Fixes test suite failures on macOS 12, arm64. See gh-14901 for details. The failures on x86-64 may not be identical, and this doesn't close the issue... — committed to tylerjereddy/scipy by rgommers 3 years ago
short-circuit overflow protection for M1 Mac clang (#827) xref scipy/scipy#14901 Splits single if statement into two to avoid apparent clang short-circuit bug on M1 Mac platforms. Co-authored-b... — committed to boostorg/math by mckib2 2 years ago

Most upvoted comments

nct should be fixed by https://github.com/boostorg/math/pull/892. I’ll look at ncx2 this week.

mborland on Dec 5, 2022

Thanks @mborland!

mdhaber on Nov 17, 2023

I believe that PR to Boost will resolve this issue, but we haven’t updated the commit of the Boost submodule in SciPy main yet, so we still see these failures in SciPy main right now. I’m about to submit a PR that silences those warnings, so I’m using this issue as a reminder to un-silence such warnings after the Boost submodule commit gets changed. We’ll close this issue when they’re all passing. Thanks for your help @mborland!

mdhaber on Oct 21, 2022

@mckib2 patch has been merged into Boost.Math. Let me know if there’s anything else that needs assistance.

mborland on Oct 12, 2022

gh-15323 should fix the failures on arm64 macOS. Not 100% sure that it fixes everything on x86_64 for macOS 12, but at least CI is already passing there. So this fix should be enough for the 1.8.0 release. I think the issue should remain open for figuring out the root cause (but that is not urgent).

rgommers on Dec 31, 2021

Nope, -O0 didn’t matter. The full test suite on macOS arm64 is now down to 13 failures, all due to overflow warnings in beta and nbinom from Boost code:

FAILED scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-beta-arg4] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-rdist-arg85] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_cont_basic[500-200-semicircular-arg89] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[beta-args4-ppf] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[beta-args4-isf] - RuntimeWarning: overflow encountered in _beta_isf
FAILED scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[rdist-args86-ppf] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[rdist-args86-isf] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[semicircular-args90-ppf] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_continuous_basic.py::test_methods_with_lists[semicircular-args90-isf] - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_discrete_basic.py::test_discrete_basic[nbinom-arg13-False] - RuntimeWarning: overflow encountered in _nbinom_ppf
FAILED scipy/stats/tests/test_discrete_basic.py::test_moments[nbinom-arg13] - RuntimeWarning: overflow encountered in _nbinom_ppf
FAILED scipy/stats/tests/test_distributions.py::TestFrozen::test_pickling - RuntimeWarning: overflow encountered in _beta_ppf
FAILED scipy/stats/tests/test_distributions.py::TestExpect::test_beta - RuntimeWarning: overflow encountered in _beta_ppf
================================ 13 failed, 35125 passed, 3080 skipped, 11454 deselected, 108 xfailed, 10 xpassed in 350.94s (0:05:50)

rgommers on Dec 26, 2021

I think CI is on Macos Intel.

andyfaff on Oct 25, 2021