scipy: BUG: sparse.linalg: Segfault in `arpack` with `ifx`
Describe your issue.
I am installing scipy from source on an HPC on top of python 3.11.6. The code builds without errors, but scipy.test() segfaults at scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py. We have intel MKL on the HPC, so I built scipy with
CC=icx CXX=icpx FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp
Reproducing Code Example
python dev.py test -s sparse
Error message
ninja: Entering directory `/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build'
[4/4] Generating scipy/generate-version with a custom command
Build OK
💻 meson install -C build --only-changed
Installing, see meson-install.log...
Installation OK
SciPy from development installed path at: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages
Running tests for scipy version:1.13.0, installed at:/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0
configfile: pytest.ini
plugins: hypothesis-6.102.4
collected 11638 items / 61 deselected / 11577 selected
scipy/sparse/csgraph/tests/test_connected_components.py ........ [ 0%]
scipy/sparse/csgraph/tests/test_conversions.py ... [ 0%]
scipy/sparse/csgraph/tests/test_flow.py ........................................... [ 0%]
scipy/sparse/csgraph/tests/test_graph_laplacian.py ......................................................................................................................... [ 1%]
............................................................................................................................................................................ [ 2%]
............................................................................................................................................................................ [ 4%]
............................................................................................................................................................................ [ 5%]
............................................................................................................................................................................ [ 7%]
............................................................................................................................................................................ [ 8%]
................................................................................................................................................. [ 10%]
scipy/sparse/csgraph/tests/test_matching.py ................................ [ 10%]
scipy/sparse/csgraph/tests/test_pydata_sparse.py ssssssssssssssssssssssssssssssssss [ 10%]
scipy/sparse/csgraph/tests/test_reordering.py ... [ 10%]
scipy/sparse/csgraph/tests/test_shortest_path.py ................................... [ 11%]
scipy/sparse/csgraph/tests/test_spanning_tree.py . [ 11%]
scipy/sparse/csgraph/tests/test_traversal.py ........ [ 11%]
scipy/sparse/linalg/_dsolve/tests/test_linsolve.py .s.s.s.s.ss..........s..................... [ 11%]
scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py .Fatal Python error: Segmentation fault
Current thread 0x0000146b7118b740 (most recent call first):
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 883 in extract
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1357 in eigs
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1577 in eigsh
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 248 in eval_evec
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 425 in test_hermitian_modes
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/_lib/_testutils.py", line 106 in __call__
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 745 in scipy_tests
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 761 in run
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/action.py", line 461 in execute
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/task.py", line 473 in execute
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 176 in execute_task
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 220 in run_tasks
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 254 in run_all
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_run.py", line 265 in _execute
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 570 in execute
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 150 in parse_execute
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/api.py", line 50 in run_tasks
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 251 in run_doit_task
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pydevtool/cli.py", line 267 in callback
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 783 in invoke
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1434 in invoke
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1688 in invoke
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/rich_click/rich_command.py", line 126 in main
File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1157 in __call__
File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 1486 in <module>
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy._lib._fpumode, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2 (total: 50)
Segmentation fault (core dumped)
SciPy/NumPy/Python version and system information
Python 3.11.6 (main, May 15 2024, 11:36:54) [GCC Intel(R) C++ gcc 11.3.1 mode] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.13.0 1.26.4 sys.version_info(major=3, minor=11, micro=6, releaselevel='final', serial=0)
/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
"Compilers": {
"c": {
"name": "intel-llvm",
"linker": "ld.bfd",
"version": "11.3.1",
"commands": "icx"
},
"cython": {
"name": "cython",
"linker": "cython",
"version": "3.0.10",
"commands": "cython"
},
"c++": {
"name": "intel-llvm",
"linker": "ld.bfd",
"version": "11.3.1",
"commands": "icpx"
},
"fortran": {
"name": "intel-llvm",
"linker": "ld.bfd",
"version": "2023.1.0",
"commands": "ifx"
},
"pythran": {
"version": "0.14.0",
"include directory": "../../../python/3.11.6/lib/python3.11/site-packages/pythran"
}
},
"Machine Information": {
"host": {
"cpu": "x86_64",
"family": "x86_64",
"endian": "little",
"system": "linux"
},
"build": {
"cpu": "x86_64",
"family": "x86_64",
"endian": "little",
"system": "linux"
},
"cross-compiled": true
},
"Build Dependencies": {
"blas": {
"name": "mkl-dynamic-lp64-iomp",
"found": true,
"version": "2023.1",
"detection method": "pkgconfig",
"include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
"lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
"openblas configuration": "unknown",
"pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
},
"lapack": {
"name": "mkl-dynamic-lp64-iomp",
"found": true,
"version": "2023.1",
"detection method": "pkgconfig",
"include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
"lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
"openblas configuration": "unknown",
"pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
},
"pybind11": {
"name": "pybind11",
"version": "2.12.0",
"detection method": "config-tool",
"include directory": "unknown"
}
},
"Python Information": {
"path": "/work2/noaa/co2/sbasu/packages/python/3.11.6/bin/python3.11",
"version": "3.11"
}
}
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 37 (36 by maintainers)
After https://github.com/czgdp1807/PROPACK/commit/b59104e306f2164deca19f6a25f98c599d4c22e8 all the segmentation faults are fixed in
python dev.py test -s sparse.I checked https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html. Seems like
ifxoptimisations can be disabled via,-O0and-fpmodel=precise. Worth trying I think.Sure. I will try it in the morning.
Yes. Now I will debug the tests. I wanted to see what all combinations work and what all fail. I have started with
ifxandMKLcombination.One weird observation with the following code (basically an MRE of the failing test),
Randomised behaviour
Results produced are correct but seems like segmentation fault is random, sometimes it happens sometimes it doesn’t. Seems like coming from LLVM generated code by
ifx.Notes
With
pdb, no segmentation fault comes up.With your PR, the segmentation fault disappears from
_arpackbut comes intest_svds,Commands
python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp(succeeds withsparsemodule)FC=ifx python dev.py build(sparsetest fail with segmentation fault)FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp(sparsetest fail with segmentation fault)FC="ifort -diag-disable=10448" python dev.py build(sparsetest fail with segmentation fault)FC="ifort -diag-disable=10448" python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp(sparsetest fail with segmentation fault)Logs
OK I bumped the ARPACK version in #21110. Maybe it can solve this issue too. No idea what the changes are as it still has issues unresolved upstream too.
Simultaneously, I started rewriting ARPACK work. The code is exactly what anyone would expect from a F77 monolith.