scipy: BUG: free-threaded 3.13 segmentation fault when calling `linalg.cho_solve` in different threads
Describe your issue.
Seen when running the scikit-learn tests with free-threaded 3.13 and reduced to a smaller snippet. With the reproducer on my machine it happens “often enough” maybe more than 50% of the time?
Right now on my machine you need a combination of three things:
- calling
cho_solvein different threads - use
threadpoolctl.threadpool_limits. UsingOPENBLAS_NUM_THREADSdoes not trigger the segfault. - create a multiprocessing Pool with the default fork method. Other fork methods do not trigger the segfault. Side-comment: the issue exists also with joblib with default (loky) backend.
Looking at the gdb bt info, you have in different threads:
dtrsm_LNLN(inbtso the one trigger the segfault?)scipy_dtrsm_in other threads
For completeness the way to trigger the segfault in scikit-learn tests:
(for i in $(seq 1 10); do PYTHON_GIL=0 pytest -v sklearn/compose/tests/test_column_transformer.py::test_column_transformer_auto_memmap sklearn/gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs; done) 2>&1 | tee free-threaded-10-times.log
Reproducing Code Example
conda create -n test-env -c conda-forge -c conda-forge/label/python_rc python-freethreading -y
# or with Anaconda testing channel
# conda create -n test-env -c defaults -c ad-testing/label/py313_nogil python=3.13 -y
conda activate test-env
pip install \
--extra-index https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
scipy threadpoolctl --only-binary :all:
PYTHONFAULTHANDLER=1 PYTHON_GIL=0 python test-simple.py
# test-simple.py
import pickle
from concurrent.futures import ThreadPoolExecutor
from multiprocessing.pool import Pool
import multiprocessing
import numpy as np
from scipy.linalg import cho_solve
# Don't ask me why but this is needed some numbers work better than other to
# trigger the segmentation fault
from threadpoolctl import threadpool_limits
threadpool_limits(10, user_api="blas")
rng = np.random.default_rng(0)
L = rng.normal(size=(50, 50))
W_sr = rng.normal(size=50)
def func(i):
cho_solve((L, True), np.diag(W_sr))
def create_multiprocessing_pool():
# does not trigger segfault
# multiprocessing.set_start_method('spawn', force=True)
# does not trigger segfault
# multiprocessing.set_start_method('forkserver', force=True)
Pool(2)
if __name__ == '__main__':
create_multiprocessing_pool()
tpe = ThreadPoolExecutor(max_workers=2)
tpe.map(func, range(10))
Error message
Python thread-level info through PYTHONFAULTHANDLER:
Fatal Python error: Segmentation fault
Thread 0x00007422702006c0 (most recent call first):
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_decomp_cholesky.py", line 218 in cho_solve
File "/tmp/test-simple.py", line 23 in func
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 58 in run
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 92 in _worker
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 992 in run
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1012 in _bootstrap
Thread 0x0000742270c006c0 (most recent call first):
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_decomp_cholesky.py", line 218 in cho_solve
File "/tmp/test-simple.py", line 23 in func
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 58 in run
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 92 in _worker
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 992 in run
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1012 in _bootstrap
Thread 0x00007422914e8640 (most recent call first):
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1092 in join
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 31 in _python_exit
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1505 in <lambda>
File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1534 in _shutdown
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, psutil._psutil_linux, psutil._psutil_posix, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering (total: 41)
[1] 2594761 segmentation fault (core dumped) PYTHONFAULTHANDLER=1 PYTHON_GIL=0 python /tmp/test-simple.py
gdb `bt` info
(gdb) bt
#0 0x000074229157f3f4 in ?? () from /usr/lib/libc.so.6
#1 0x0000742291526120 in raise () from /usr/lib/libc.so.6
#2 <signal handler called>
#3 0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4 0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5 0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6 0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#7 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#8 0x000074229160249c in ?? () from /usr/lib/libc.so.6
`gdb apply all bt` info
(gdb) thread apply all bt
Thread 14 (Thread 0x742253e006c0 (LWP 2594803)):
#0 0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2 0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3 0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#5 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 13 (Thread 0x74226f8006c0 (LWP 2594794)):
#0 0x000074226d5b1a40 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2 0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#4 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 12 (Thread 0x742270c006c0 (LWP 2594792)):
#0 0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2 0x000074226d6fa8d8 in exec_blas.part () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3 0x000074226d6fa9a0 in exec_blas () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4 0x000074226d6f84e1 in gemm_thread_n () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5 0x000074226d4c9f1a in scipy_dtrsm_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6 0x000074226dfaebb9 in scipy_dpotrs_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#7 0x000074224e274c78 in f2py_rout.flapack_dpotrs () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_flapack.cpython-313t-x86_64-linux-gnu.so
#8 0x000000000050f5cc in _PyObject_MakeTpCall (tstate=0xe85e20, callable=0x2fa99c74840, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:242
#9 0x0000000000526c18 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/generated_cases.c.h:1500
#10 0x000000000055d63c in _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x742270bffe18, func=0x2fa98b5f8c0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:413
#11 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x742270bffe18, callable=0x2fa98b5f8c0, tstate=0xe85e20) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#12 method_vectorcall (method=<optimized out>, args=0x886620 <_PyRuntime+112352>, nargsf=0, kwnames=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/classobject.c:70
#13 0x000000000063cc9b in thread_run (boot_raw=0xe85de0) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:337
#14 0x00000000005f5624 in pythread_wrapper (arg=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/thread_pthread.h:243
#15 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#16 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 11 (Thread 0x7422914e8640 (LWP 2594761)):
#0 0x0000742291579a19 in ?? () from /usr/lib/libc.so.6
#1 0x0000742291585858 in ?? () from /usr/lib/libc.so.6
#2 0x000000000056dd38 in _PySemaphore_PlatformWait (timeout=-1, sema=0x7ffe1fe31970) at /usr/local/src/conda/python-3.13.0rc1/Python/parking_lot.c:142
#3 _PySemaphore_Wait (sema=0x7ffe1fe31970, timeout=-1, detach=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/parking_lot.c:213
#4 0x00000000005e1772 in _PyParkingLot_Park (addr=addr@entry=0xe66939, expected=expected@entry=0x7ffe1fe319f7, size=size@entry=1, timeout_ns=timeout_ns@entry=-1, park_arg=park_arg@entry=0x0, detach=detach@entry=1) at /usr/local/src/conda/python-3.13.0rc1/Python/parking_lot.c:316
#5 0x000000000063cd79 in PyEvent_WaitTimed (evt=evt@entry=0xe66939, timeout_ns=timeout_ns@entry=-1, detach=detach@entry=1) at /usr/local/src/conda/python-3.13.0rc1/Python/lock.c:290
#6 0x000000000063d4f8 in ThreadHandle_join (self=0xe66900, timeout_ns=-1) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:505
#7 0x000000000063d1fc in PyThreadHandleObject_join (self=0x2fa9ace39f0, args=args@entry=0x2fa9ace4020) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:642
#8 0x00000000005aeb0a in method_vectorcall_VARARGS (func=<optimized out>, args=0x7422917fe200, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.13.0rc1/Objects/descrobject.c:324
#9 0x0000000000510e24 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=0x7422917fe200, callable=0x2fa983439d0, tstate=0x8b6b70 <_PyRuntime+310320>) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#10 PyObject_Vectorcall (callable=0x2fa983439d0, args=0x7422917fe200, nargsf=9223372036854775810, kwnames=0x0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:327
#11 0x00000000005223cd in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/generated_cases.c.h:813
#12 0x0000000000515740 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffe1fe31cf8, callable=0x2fa98e51220, tstate=0x8b6b70 <_PyRuntime+310320>) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#13 PyObject_VectorcallMethod (name=name@entry=0x879530 <_PyRuntime+58864>, args=0x7ffe1fe31cf8, args@entry=0x7ffe1fe31cf0, nargsf=<optimized out>, nargsf@entry=9223372036854775809, kwnames=kwnames@entry=0x0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:856
#14 0x00000000005f55ac in PyObject_CallMethodNoArgs (name=0x879530 <_PyRuntime+58864>, self=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Include/cpython/abstract.h:65
#15 wait_for_thread_shutdown (tstate=0x8b6b70 <_PyRuntime+310320>) at /usr/local/src/conda/python-3.13.0rc1/Python/pylifecycle.c:3435
#16 0x00000000005da5e0 in _Py_Finalize (runtime=0x86af40 <_PyRuntime>) at /usr/local/src/conda/python-3.13.0rc1/Python/pylifecycle.c:2080
#17 0x00000000005fd115 in Py_FinalizeEx () at /usr/local/src/conda/python-3.13.0rc1/Python/pylifecycle.c:2302
#18 0x00000000005f065f in Py_RunMain () at /usr/local/src/conda/python-3.13.0rc1/Modules/main.c:777
#19 0x00000000005965e9 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Modules/main.c:829
#20 0x000074229150ee08 in ?? () from /usr/lib/libc.so.6
#21 0x000074229150eecc in __libc_start_main () from /usr/lib/libc.so.6
#22 0x0000000000595970 in _start ()
Thread 10 (Thread 0x7422548006c0 (LWP 2594802)):
#0 0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
--Type <RET> for more, q to quit, c to continue without paging--c
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 9 (Thread 0x7422552006c0 (LWP 2594801)):
#0 0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 8 (Thread 0x7422720006c0 (LWP 2594800)):
#0 0x000074226d6fa214 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 7 (Thread 0x742272a006c0 (LWP 2594799)):
#0 0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 6 (Thread 0x7422734006c0 (LWP 2594798)):
#0 0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 5 (Thread 0x742273e006c0 (LWP 2594797)):
#0 0x000074226d6fa214 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 4 (Thread 0x7422748006c0 (LWP 2594796)):
#0 0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 3 (Thread 0x7422716006c0 (LWP 2594795)):
#0 0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 2 (Thread 0x7422702006c0 (LWP 2594793)):
#0 0x000074226d6fa7fe in exec_blas_async_wait () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1 0x000074226d6fa8f0 in exec_blas.part () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2 0x000074226d6fa9a0 in exec_blas () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3 0x000074226d6f84e1 in gemm_thread_n () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4 0x000074226d4c9f1a in scipy_dtrsm_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5 0x000074226dfaebb9 in scipy_dpotrs_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6 0x000074224e274c78 in f2py_rout.flapack_dpotrs () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_flapack.cpython-313t-x86_64-linux-gnu.so
#7 0x000000000050f5cc in _PyObject_MakeTpCall (tstate=0xe8a3c0, callable=0x2fa99c74840, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:242
#8 0x0000000000526c18 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/generated_cases.c.h:1500
#9 0x000000000055d63c in _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x7422701ffe18, func=0x2fa98b5f8c0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:413
#10 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7422701ffe18, callable=0x2fa98b5f8c0, tstate=0xe8a3c0) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#11 method_vectorcall (method=<optimized out>, args=0x886620 <_PyRuntime+112352>, nargsf=0, kwnames=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/classobject.c:70
#12 0x000000000063cc9b in thread_run (boot_raw=0xe85da0) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:337
#13 0x00000000005f5624 in pythread_wrapper (arg=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/thread_pthread.h:243
#14 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#15 0x000074229160249c in ?? () from /usr/lib/libc.so.6
Thread 1 (Thread 0x7422534006c0 (LWP 2594804)):
#0 0x000074229157f3f4 in ?? () from /usr/lib/libc.so.6
#1 0x0000742291526120 in raise () from /usr/lib/libc.so.6
#2 <signal handler called>
#3 0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4 0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5 0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6 0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#7 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#8 0x000074229160249c in ?? () from /usr/lib/libc.so.6
SciPy/NumPy/Python version and system information
{
"Compilers": {
"c": {
"name": "gcc",
"linker": "ld.bfd",
"version": "10.2.1",
"commands": "cc"
},
"cython": {
"name": "cython",
"linker": "cython",
"version": "3.1.0",
"commands": "cython"
},
"c++": {
"name": "gcc",
"linker": "ld.bfd",
"version": "10.2.1",
"commands": "c++"
},
"fortran": {
"name": "gcc",
"linker": "ld.bfd",
"version": "10.2.1",
"commands": "gfortran"
},
"pythran": {
"version": "0.16.1",
"include directory": "../../opt/python/cp313-cp313t/lib/python3.13t/site-packages/pythran"
}
},
"Machine Information": {
"host": {
"cpu": "x86_64",
"family": "x86_64",
"endian": "little",
"system": "linux"
},
"build": {
"cpu": "x86_64",
"family": "x86_64",
"endian": "little",
"system": "linux"
},
"cross-compiled": false
},
"Build Dependencies": {
"blas": {
"name": "scipy-openblas",
"found": true,
"version": "0.3.28",
"detection method": "pkgconfig",
"include directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/include",
"lib directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/lib",
"openblas configuration": "OpenBLAS 0.3.28 DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64",
"pc file directory": "/project"
},
"lapack": {
"name": "scipy-openblas",
"found": true,
"version": "0.3.28",
"detection method": "pkgconfig",
"include directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/include",
"lib directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/lib",
"openblas configuration": "OpenBLAS 0.3.28 DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64",
"pc file directory": "/project"
},
"pybind11": {
"name": "pybind11",
"version": "2.13.4",
"detection method": "config-tool",
"include directory": "unknown"
}
},
"Python Information": {
"path": "/opt/python/cp313-cp313t/bin/python",
"version": "3.13"
}
}
cc @betatim (to satisfy his curiosity)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 21 (21 by maintainers)
@lesteve I have successfully reproduced the segfaults on my Ubuntu 22.04.4 machine following your instructions. So it isn’t just your machine!
I build OpenBLAS dev version locally with and the problem has gone away. I guess this is enough to close this one. Wow this feels like the end of quite a journey 😉.
The make command I used:
and then copied the .so to the pixi env (not sure if there is a cleaner way 🤷)
I can not reproduce the issue with (I ran it 20 times to be sure):
I am reasonably confident that the reason we have not seen it in the scikit-learn CI is because the CI does not thoroughly test this kind of nested parallelism. We have 2 cores and we do
pytest -n2(see https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=72333&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=ef785ae2-496b-5b02-9f0e-07a6c3ab3081&l=176) so BLAS is single-threaded and we don’t trigger this bug.Note that https://github.com/scikit-learn/scikit-learn/issues/30151 is a segmentation fault that can be triggered by scikit-learn usage using regular CPython (without using the experimental free-threading interpreter) but in a specific context that implies chaining several seemingly unrelated pre-conditions (see the minimal reproducer in the issue for details).
If scipy 1.15.0 ships OpenBLAS 0.3.28 it might raise the likelihood of triggering this crash. However, running the full scikit-learn test suite with the nightly build of scipy 1.15.0.dev0 that currently ships 0penBLAS 0.3.28 cannot trigger the crash on our CI:
https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=72333&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=ef785ae2-496b-5b02-9f0e-07a6c3ab3081&l=40
So maybe the required conditions to trigger the problem are rare enough that this regression would not impact many people in practice.
I reduced the test case to this:
The
threadpool_limitscall is indeed needed - with it the crash happens (thread limit can be 10, or 4 or 20, doesn’t matter), without it the test reliably passes, even with another layer of threads on top withpytest-run-parallel.I also tested with MKL and Netlib BLAS/LAPACK on Linux, there is no problem there. The problem seems localized to
threadpoolctl+ OpenBLAS.I also made it easier to set up a free-threading environment to reproduce problems like this with Pixi:
git clone https://github.com/rgommers/pixi-dev-scipystack/cd pixi-dev-scipystack/scipy && git clone https://github.com/rgommers/scipycd scipy && git checkout cho-solve-crash(thischo-solve-crashbranch)If you want to use
pytest-run-parallelon top:If you want to switch from OpenBLAS to MKL or Netlib, edit the
pixi.tomlinpixi-dev-scipystack/scipyand add themklornetlibfeature to thefree-threadingenvironment and then re-run thepixi r test-nogil ...command. E.g.:Thanks for the links @lucascolley, looks like this is one way to install free-threaded CPython from conda-forge: