scipy: BUG: Test failures due to `invalid value encountered in _beta_ppf` on M2 mac

Describe your issue.

The following tests fail with the warning invalid value encountered in _beta_ppf

Reproducing Code Example

`python dev.py test`

Error message

FAILED scipy/stats/tests/test_distributions.py::TestBeta::test_issue_12635 - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_distributions.py::TestBeta::test_issue_12794 - RuntimeWarning: invalid value encountered in _beta_isf
FAILED scipy/stats/tests/test_distributions.py::TestBeta::test_issue_12796 - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fast_gen_inversion.py::test_rvs_and_ppf[betaprime-args6] - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_distributions.py::TestBetaPrime::test_ppf[0.375-0.25-7.0-0.002036820346115211] - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fast_gen_inversion.py::test_non_rvs_methods_with_domain - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fit.py::TestGoodnessOfFit::test_against_filliben_norm - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fit.py::TestGoodnessOfFit::test_filliben_property - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fit.py::TestGoodnessOfFit::test_against_filliben_norm_table[case0] - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fit.py::TestGoodnessOfFit::test_against_filliben_norm_table[case1] - RuntimeWarning: invalid value encountered in _beta_ppf
FAILED scipy/stats/tests/test_fit.py::TestGoodnessOfFit::test_against_filliben_norm_table[case2] - RuntimeWarning: invalid value encountered in _beta_ppf

SciPy/NumPy/Python version and system information

1.13.0.dev0+0.9979b1d 1.26.4 sys.version_info(major=3, minor=11, micro=8, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /Users/kai/miniconda3/envs/scipy-dev/include
    lib directory: /Users/kai/miniconda3/envs/scipy-dev/lib
    name: openblas
    openblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=0 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK=0 NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP=1 VORTEX MAX_THREADS=128
    pc file directory: /Users/kai/miniconda3/envs/scipy-dev/lib/pkgconfig
    version: 0.3.26
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /Users/kai/miniconda3/envs/scipy-dev/include
    lib directory: /Users/kai/miniconda3/envs/scipy-dev/lib
    name: openblas
    openblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=0 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK=0 NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP=1 VORTEX MAX_THREADS=128
    pc file directory: /Users/kai/miniconda3/envs/scipy-dev/lib/pkgconfig
    version: 0.3.26
  pybind11:
    detection method: pkgconfig
    include directory: /Users/kai/miniconda3/envs/scipy-dev/include
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    args: -ftree-vectorize, -fPIC, -fstack-protector-strong, -O2, -pipe, -isystem,
      /Users/kai/miniconda3/envs/scipy-dev/include, -D_FORTIFY_SOURCE=2, -isystem,
      /Users/kai/miniconda3/envs/scipy-dev/include
    commands: arm64-apple-darwin20.0.0-clang
    linker: ld64
    linker args: -Wl,-headerpad_max_install_names, -Wl,-dead_strip_dylibs, -Wl,-rpath,/Users/kai/miniconda3/envs/scipy-dev/lib,
      -L/Users/kai/miniconda3/envs/scipy-dev/lib, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -O2, -pipe, -isystem, /Users/kai/miniconda3/envs/scipy-dev/include, -D_FORTIFY_SOURCE=2,
      -isystem, /Users/kai/miniconda3/envs/scipy-dev/include
    name: clang
    version: 16.0.6
  c++:
    args: -ftree-vectorize, -fPIC, -fstack-protector-strong, -O2, -pipe, -stdlib=libc++,
      -fvisibility-inlines-hidden, -fmessage-length=0, -isystem, /Users/kai/miniconda3/envs/scipy-dev/include,
      -D_FORTIFY_SOURCE=2, -isystem, /Users/kai/miniconda3/envs/scipy-dev/include
    commands: arm64-apple-darwin20.0.0-clang++
    linker: ld64
    linker args: -Wl,-headerpad_max_install_names, -Wl,-dead_strip_dylibs, -Wl,-rpath,/Users/kai/miniconda3/envs/scipy-dev/lib,
      -L/Users/kai/miniconda3/envs/scipy-dev/lib, -ftree-vectorize, -fPIC, -fstack-protector-strong,
      -O2, -pipe, -stdlib=libc++, -fvisibility-inlines-hidden, -fmessage-length=0,
      -isystem, /Users/kai/miniconda3/envs/scipy-dev/include, -D_FORTIFY_SOURCE=2,
      -isystem, /Users/kai/miniconda3/envs/scipy-dev/include
    name: clang
    version: 16.0.6
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.9
  fortran:
    args: -march=armv8.3-a, -ftree-vectorize, -fPIC, -fno-stack-protector, -O2, -pipe,
      -isystem, /Users/kai/miniconda3/envs/scipy-dev/include
    commands: /Users/kai/miniconda3/envs/scipy-dev/bin/arm64-apple-darwin20.0.0-gfortran
    linker: ld64
    linker args: -Wl,-headerpad_max_install_names, -Wl,-dead_strip_dylibs, -Wl,-rpath,/Users/kai/miniconda3/envs/scipy-dev/lib,
      -L/Users/kai/miniconda3/envs/scipy-dev/lib, -march=armv8.3-a, -ftree-vectorize,
      -fPIC, -fno-stack-protector, -O2, -pipe, -isystem, /Users/kai/miniconda3/envs/scipy-dev/include
    name: gcc
    version: 12.3.0
  pythran:
    include directory: ../../../miniconda3/envs/scipy-dev/lib/python3.11/site-packages/pythran
    version: 0.15.0
Machine Information:
  build:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
  cross-compiled: false
  host:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
Python Information:
  path: /Users/kai/miniconda3/envs/scipy-dev/bin/python3.11
  version: '3.11'


</details>

About this issue

Original URL
State: open
Created 4 months ago
Comments: 19 (19 by maintainers)

Most upvoted comments

Ah yeah, and I bet you’re both using conda too.

tylerjereddy on Mar 8, 2024

So may be for now we should fix compilers=1.6.0 in environment.yml file

We can live with these failures for a little while, since they don’t show up in CI. Let’s try to resolve it first before pinning. It’s not clearcut that it’s only a Clang 16 bug; there are fixes in boost-math that we haven’t pulled in yet, and there may be another bug (or multiple) hiding in boost-math.

and also upgrade boost/math to the latest commit.

Could you have a go at this?

rgommers on Mar 29, 2024

I will start looking into it in the afternoon today.

czgdp1807 on Mar 28, 2024

Those tests are passing on my M2 with Apple clang version 15.0.0 (clang-1500.3.9.4). Are you using the newer clang from above as well Lucas?

tylerjereddy on Mar 8, 2024

We could try to replace the custom Boost wrappers for the beta distribution by directly calling the functions from special then. The required ones are

That would also reduce duplication between stats and special. special on its own has enough duplication already 😉.

Would you be able to test it out? I do not have an M2 Mac to test locally.

dschmitz89 on Mar 8, 2024

Do you get the same errors using the incomplete beta function via special? That one also uses boost under the hood but was wrapped in a different way, might be worth trying out.

dschmitz89 on Mar 7, 2024