scipy: BUG: 1.10.0rc2 Windows wheel tests runs all segfault
The wheel build jobs are here: https://github.com/scipy/scipy/actions/runs/3753070381
The non-informative failures were originally discussed here, and Ralf and I kind of went for trying to build the wheels anyway, because it wasn’t clear there was a genuine SciPy issue: https://github.com/scipy/scipy/pull/17640#issuecomment-1362081751
We have just slightly more information from the cibuildwheel testing now, but not too much more–we now know it is a segfault:
D:\a\scipy\scipy/tools/wheels/cibw_test_command.sh: line 11: 1391 Segmentation fault python -c "import sys; import scipy; sys.exit(not scipy.test())"
For now, I’m tentatively labeling this is an upstream bug since it started with NumPy 1.24.0.
Two solutions I can think of:
- try to identify the problematic test and skip it with NumPy >= 1.24.0 pending further investigation, on Windows only
- (easier, I think?): pin the runtime testing for Windows wheel builds on GitHub actions to NumPy < 1.24.0 temporarily
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 44 (38 by maintainers)
Commits related to this issue
- MAINT: pybind11 win exclusion Fixes #17644 (see detailed analysis over there..) * I confirmed locally that `cibuildwheel --platform windows` passes with `pybind11` `2.10.2` excluded, whereas we curr... — committed to tylerjereddy/scipy by tylerjereddy 2 years ago
- MAINT: pybind11 win exclusion Fixes #17644 (see detailed analysis over there..) * I confirmed locally that `cibuildwheel --platform windows` passes with `pybind11` `2.10.2` excluded, whereas we curr... — committed to tylerjereddy/scipy by tylerjereddy 2 years ago
- MAINT: pybind11 win exclusion Fixes #17644 (see detailed analysis over there..) * I confirmed locally that `cibuildwheel --platform windows` passes with `pybind11` `2.10.2` excluded, whereas we curr... — committed to windows-server-2003/scipy by tylerjereddy 2 years ago
Ok. Figured this out. scipy has a bug here in that it is calling Python functions before Python is initialized. This is undefined behavior and in general quite bad. On pybind11’s side, we just started trying to detect that error case, but we should probably make the error logging much better here (we were throwing an exception, but that exception gets swallowed during the DLL load process).
I’ll submit a PR in an hour or so here to fix the bug within scipy.
I’ll separately submit a PR to pybind11 to improve the logging for this edge case.
CC @rwgk
I excised the reproducer from SciPy in case it helps: https://github.com/tylerjereddy/pybind_repro
Basically one
meson.buildfile, and C++ source + header is sufficient. That probably cuts down the iteration times a bit at least.@rwgk I can take over debugging if you wanr, since it’s likely my change and I have a Windows machine handy.
I think this guide to building on Windows is more up-to-date: https://github.com/scipy/scipy/wiki/Easy-Windows-10-Build-Scipy-from-source . @rgommers - is it up-to-date on OpenBLAS binaries?
@Lalaland Basically, OpenBLAS is(was) the annoying step back in the day with OpenBLAS not being available so we went in that rabbit hole in those instructions. However now you can download OpenBLAS from https://anaconda.org/multibuild-wheels-staging/openblas-libs/files . Once you have it, install MinGW64 and add the bin folder to the path so that you have a working gcc and gfortran on the path. Then make an environment variable
(wherever you downloaded the OpenBLAS to) pointing to the OpenBLAS. Then you can use
to build. If you encounter pythran issues also add the environment variable
There is also #17657 for this step but we can address that when we get there.
If it’s something we need to fix, happy to push a release.
Yup, excluding that version of
pybind11did the trick, related PR/issues cross-linked above.