scipy: BUG: free-threaded 3.13 segmentation fault when calling `linalg.cho_solve` in different threads

Describe your issue.

Seen when running the scikit-learn tests with free-threaded 3.13 and reduced to a smaller snippet. With the reproducer on my machine it happens “often enough” maybe more than 50% of the time?

Right now on my machine you need a combination of three things:

  • calling cho_solve in different threads
  • use threadpoolctl.threadpool_limits. Using OPENBLAS_NUM_THREADS does not trigger the segfault.
  • create a multiprocessing Pool with the default fork method. Other fork methods do not trigger the segfault. Side-comment: the issue exists also with joblib with default (loky) backend.

Looking at the gdb bt info, you have in different threads:

  • dtrsm_LNLN (in bt so the one trigger the segfault?)
  • scipy_dtrsm_ in other threads

For completeness the way to trigger the segfault in scikit-learn tests:

(for i in $(seq 1 10); do PYTHON_GIL=0 pytest -v sklearn/compose/tests/test_column_transformer.py::test_column_transformer_auto_memmap sklearn/gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs; done) 2>&1 | tee free-threaded-10-times.log

Reproducing Code Example

conda create -n test-env -c conda-forge -c conda-forge/label/python_rc python-freethreading -y
# or with Anaconda testing channel
# conda create -n test-env -c defaults -c ad-testing/label/py313_nogil python=3.13 -y
conda activate test-env
pip install \
    --extra-index https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
    scipy threadpoolctl --only-binary :all:
PYTHONFAULTHANDLER=1 PYTHON_GIL=0 python test-simple.py
# test-simple.py
import pickle
from concurrent.futures import ThreadPoolExecutor
from multiprocessing.pool import Pool
import multiprocessing

import numpy as np

from scipy.linalg import cho_solve

# Don't ask me why but this is needed some numbers work better than other to
# trigger the segmentation fault
from threadpoolctl import threadpool_limits
threadpool_limits(10, user_api="blas")


rng = np.random.default_rng(0)
L = rng.normal(size=(50, 50))
W_sr = rng.normal(size=50)

def func(i):
    cho_solve((L, True), np.diag(W_sr))

def create_multiprocessing_pool():
    # does not trigger segfault
    # multiprocessing.set_start_method('spawn', force=True)
    # does not trigger segfault
    # multiprocessing.set_start_method('forkserver', force=True)
    Pool(2)


if __name__ == '__main__':
    create_multiprocessing_pool()

    tpe = ThreadPoolExecutor(max_workers=2)
    tpe.map(func, range(10))

Error message

Python thread-level info through PYTHONFAULTHANDLER:

Fatal Python error: Segmentation fault

Thread 0x00007422702006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_decomp_cholesky.py", line 218 in cho_solve
  File "/tmp/test-simple.py", line 23 in func
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 58 in run
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 92 in _worker
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 992 in run
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x0000742270c006c0 (most recent call first):
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_decomp_cholesky.py", line 218 in cho_solve
  File "/tmp/test-simple.py", line 23 in func
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 58 in run
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 92 in _worker
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 992 in run
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x00007422914e8640 (most recent call first):
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1092 in join
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/concurrent/futures/thread.py", line 31 in _python_exit
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1505 in <lambda>
  File "/home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/threading.py", line 1534 in _shutdown

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, psutil._psutil_linux, psutil._psutil_posix, cython.cimports.libc.math, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering (total: 41)
[1]    2594761 segmentation fault (core dumped)  PYTHONFAULTHANDLER=1 PYTHON_GIL=0 python /tmp/test-simple.py
gdb `bt` info
(gdb) bt
#0  0x000074229157f3f4 in ?? () from /usr/lib/libc.so.6
#1  0x0000742291526120 in raise () from /usr/lib/libc.so.6
#2  <signal handler called>
#3  0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4  0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5  0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6  0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#7  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#8  0x000074229160249c in ?? () from /usr/lib/libc.so.6
`gdb apply all bt` info
(gdb) thread apply all bt

Thread 14 (Thread 0x742253e006c0 (LWP 2594803)):
#0  0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2  0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3  0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#5  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 13 (Thread 0x74226f8006c0 (LWP 2594794)):
#0  0x000074226d5b1a40 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2  0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#4  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 12 (Thread 0x742270c006c0 (LWP 2594792)):
#0  0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2  0x000074226d6fa8d8 in exec_blas.part () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3  0x000074226d6fa9a0 in exec_blas () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4  0x000074226d6f84e1 in gemm_thread_n () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5  0x000074226d4c9f1a in scipy_dtrsm_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6  0x000074226dfaebb9 in scipy_dpotrs_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#7  0x000074224e274c78 in f2py_rout.flapack_dpotrs () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_flapack.cpython-313t-x86_64-linux-gnu.so
#8  0x000000000050f5cc in _PyObject_MakeTpCall (tstate=0xe85e20, callable=0x2fa99c74840, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:242
#9  0x0000000000526c18 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/generated_cases.c.h:1500
#10 0x000000000055d63c in _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x742270bffe18, func=0x2fa98b5f8c0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:413
#11 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x742270bffe18, callable=0x2fa98b5f8c0, tstate=0xe85e20) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#12 method_vectorcall (method=<optimized out>, args=0x886620 <_PyRuntime+112352>, nargsf=0, kwnames=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/classobject.c:70
#13 0x000000000063cc9b in thread_run (boot_raw=0xe85de0) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:337
#14 0x00000000005f5624 in pythread_wrapper (arg=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/thread_pthread.h:243
#15 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#16 0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 11 (Thread 0x7422914e8640 (LWP 2594761)):
#0  0x0000742291579a19 in ?? () from /usr/lib/libc.so.6
#1  0x0000742291585858 in ?? () from /usr/lib/libc.so.6
#2  0x000000000056dd38 in _PySemaphore_PlatformWait (timeout=-1, sema=0x7ffe1fe31970) at /usr/local/src/conda/python-3.13.0rc1/Python/parking_lot.c:142
#3  _PySemaphore_Wait (sema=0x7ffe1fe31970, timeout=-1, detach=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/parking_lot.c:213
#4  0x00000000005e1772 in _PyParkingLot_Park (addr=addr@entry=0xe66939, expected=expected@entry=0x7ffe1fe319f7, size=size@entry=1, timeout_ns=timeout_ns@entry=-1, park_arg=park_arg@entry=0x0, detach=detach@entry=1) at /usr/local/src/conda/python-3.13.0rc1/Python/parking_lot.c:316
#5  0x000000000063cd79 in PyEvent_WaitTimed (evt=evt@entry=0xe66939, timeout_ns=timeout_ns@entry=-1, detach=detach@entry=1) at /usr/local/src/conda/python-3.13.0rc1/Python/lock.c:290
#6  0x000000000063d4f8 in ThreadHandle_join (self=0xe66900, timeout_ns=-1) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:505
#7  0x000000000063d1fc in PyThreadHandleObject_join (self=0x2fa9ace39f0, args=args@entry=0x2fa9ace4020) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:642
#8  0x00000000005aeb0a in method_vectorcall_VARARGS (func=<optimized out>, args=0x7422917fe200, nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.13.0rc1/Objects/descrobject.c:324
#9  0x0000000000510e24 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=0x7422917fe200, callable=0x2fa983439d0, tstate=0x8b6b70 <_PyRuntime+310320>) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#10 PyObject_Vectorcall (callable=0x2fa983439d0, args=0x7422917fe200, nargsf=9223372036854775810, kwnames=0x0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:327
#11 0x00000000005223cd in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/generated_cases.c.h:813
#12 0x0000000000515740 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffe1fe31cf8, callable=0x2fa98e51220, tstate=0x8b6b70 <_PyRuntime+310320>) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#13 PyObject_VectorcallMethod (name=name@entry=0x879530 <_PyRuntime+58864>, args=0x7ffe1fe31cf8, args@entry=0x7ffe1fe31cf0, nargsf=<optimized out>, nargsf@entry=9223372036854775809, kwnames=kwnames@entry=0x0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:856
#14 0x00000000005f55ac in PyObject_CallMethodNoArgs (name=0x879530 <_PyRuntime+58864>, self=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Include/cpython/abstract.h:65
#15 wait_for_thread_shutdown (tstate=0x8b6b70 <_PyRuntime+310320>) at /usr/local/src/conda/python-3.13.0rc1/Python/pylifecycle.c:3435
#16 0x00000000005da5e0 in _Py_Finalize (runtime=0x86af40 <_PyRuntime>) at /usr/local/src/conda/python-3.13.0rc1/Python/pylifecycle.c:2080
#17 0x00000000005fd115 in Py_FinalizeEx () at /usr/local/src/conda/python-3.13.0rc1/Python/pylifecycle.c:2302
#18 0x00000000005f065f in Py_RunMain () at /usr/local/src/conda/python-3.13.0rc1/Modules/main.c:777
#19 0x00000000005965e9 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Modules/main.c:829
#20 0x000074229150ee08 in ?? () from /usr/lib/libc.so.6
#21 0x000074229150eecc in __libc_start_main () from /usr/lib/libc.so.6
#22 0x0000000000595970 in _start ()

Thread 10 (Thread 0x7422548006c0 (LWP 2594802)):
#0  0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
--Type <RET> for more, q to quit, c to continue without paging--c
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 9 (Thread 0x7422552006c0 (LWP 2594801)):
#0  0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 8 (Thread 0x7422720006c0 (LWP 2594800)):
#0  0x000074226d6fa214 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 7 (Thread 0x742272a006c0 (LWP 2594799)):
#0  0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 6 (Thread 0x7422734006c0 (LWP 2594798)):
#0  0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 5 (Thread 0x742273e006c0 (LWP 2594797)):
#0  0x000074226d6fa214 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 4 (Thread 0x7422748006c0 (LWP 2594796)):
#0  0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7422716006c0 (LWP 2594795)):
#0  0x000074226d6fa227 in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#2  0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7422702006c0 (LWP 2594793)):
#0  0x000074226d6fa7fe in exec_blas_async_wait () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#1  0x000074226d6fa8f0 in exec_blas.part () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#2  0x000074226d6fa9a0 in exec_blas () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#3  0x000074226d6f84e1 in gemm_thread_n () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4  0x000074226d4c9f1a in scipy_dtrsm_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5  0x000074226dfaebb9 in scipy_dpotrs_ () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6  0x000074224e274c78 in f2py_rout.flapack_dpotrs () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/_flapack.cpython-313t-x86_64-linux-gnu.so
#7  0x000000000050f5cc in _PyObject_MakeTpCall (tstate=0xe8a3c0, callable=0x2fa99c74840, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:242
#8  0x0000000000526c18 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/generated_cases.c.h:1500
#9  0x000000000055d63c in _PyFunction_Vectorcall (kwnames=0x0, nargsf=1, stack=0x7422701ffe18, func=0x2fa98b5f8c0) at /usr/local/src/conda/python-3.13.0rc1/Objects/call.c:413
#10 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7422701ffe18, callable=0x2fa98b5f8c0, tstate=0xe8a3c0) at /usr/local/src/conda/python-3.13.0rc1/Include/internal/pycore_call.h:168
#11 method_vectorcall (method=<optimized out>, args=0x886620 <_PyRuntime+112352>, nargsf=0, kwnames=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Objects/classobject.c:70
#12 0x000000000063cc9b in thread_run (boot_raw=0xe85da0) at /usr/local/src/conda/python-3.13.0rc1/Modules/_threadmodule.c:337
#13 0x00000000005f5624 in pythread_wrapper (arg=<optimized out>) at /usr/local/src/conda/python-3.13.0rc1/Python/thread_pthread.h:243
#14 0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#15 0x000074229160249c in ?? () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7422534006c0 (LWP 2594804)):
#0  0x000074229157f3f4 in ?? () from /usr/lib/libc.so.6
#1  0x0000742291526120 in raise () from /usr/lib/libc.so.6
#2  <signal handler called>
#3  0x000074226db88394 in dtrsm_iltncopy_HASWELL () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#4  0x000074226d5b1ba9 in dtrsm_LNLN () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#5  0x000074226d6fa085 in exec_threads () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#6  0x000074226d6fa2ac in blas_thread_server () from /home/lesteve/micromamba/envs/free-threaded-3.13/lib/python3.13t/site-packages/scipy/linalg/../../scipy.libs/libscipy_openblas-68440149.so
#7  0x000074229157d39d in ?? () from /usr/lib/libc.so.6
#8  0x000074229160249c in ?? () from /usr/lib/libc.so.6

SciPy/NumPy/Python version and system information

{
  "Compilers": {
    "c": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "cc"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.1.0",
      "commands": "cython"
    },
    "c++": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "c++"
    },
    "fortran": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.2.1",
      "commands": "gfortran"
    },
    "pythran": {
      "version": "0.16.1",
      "include directory": "../../opt/python/cp313-cp313t/lib/python3.13t/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "cross-compiled": false
  },
  "Build Dependencies": {
    "blas": {
      "name": "scipy-openblas",
      "found": true,
      "version": "0.3.28",
      "detection method": "pkgconfig",
      "include directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/include",
      "lib directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/lib",
      "openblas configuration": "OpenBLAS 0.3.28 DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64",
      "pc file directory": "/project"
    },
    "lapack": {
      "name": "scipy-openblas",
      "found": true,
      "version": "0.3.28",
      "detection method": "pkgconfig",
      "include directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/include",
      "lib directory": "/opt/_internal/cpython-3.13.0rc1-nogil/lib/python3.13t/site-packages/scipy_openblas32/lib",
      "openblas configuration": "OpenBLAS 0.3.28 DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64",
      "pc file directory": "/project"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.13.4",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/opt/python/cp313-cp313t/bin/python",
    "version": "3.13"
  }
}

cc @betatim (to satisfy his curiosity)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 21 (21 by maintainers)

Commits related to this issue

Most upvoted comments

@lesteve I have successfully reproduced the segfaults on my Ubuntu 22.04.4 machine following your instructions. So it isn’t just your machine!

The OpenBLAS issue you linked is already fixed, so I guess we can test now if that resolves the problem.

I build OpenBLAS dev version locally with and the problem has gone away. I guess this is enough to close this one. Wow this feels like the end of quite a journey 😉.

The make command I used:

make shared QUIET_MAKE=0 DYNAMIC_ARCH=1 BINARY=64 NO_LAPACK=0 TARGET=PRESCOTT NO_AFFINITY=1 NUM_THREADS=128

and then copied the .so to the pixi env (not sure if there is a cleaner way 🤷)

cp ~/dev/OpenBLAS/libopenblasp-r0.3.28.dev.so .pixi/envs/free-threading/lib/libopenblasp-r0.3.28.so 

I can not reproduce the issue with (I ran it 20 times to be sure):

pixi r test-nogil -t scipy.linalg.tests.test_cho_solve_threading -- --parallel-threads=10

However, running the full scikit-learn test suite with the nightly build of scipy 1.15.0.dev0 that currently ships 0penBLAS 0.3.28 cannot trigger the crash on our CI:

I am reasonably confident that the reason we have not seen it in the scikit-learn CI is because the CI does not thoroughly test this kind of nested parallelism. We have 2 cores and we do pytest -n2 (see https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=72333&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=ef785ae2-496b-5b02-9f0e-07a6c3ab3081&l=176) so BLAS is single-threaded and we don’t trigger this bug.

Note that https://github.com/scikit-learn/scikit-learn/issues/30151 is a segmentation fault that can be triggered by scikit-learn usage using regular CPython (without using the experimental free-threading interpreter) but in a specific context that implies chaining several seemingly unrelated pre-conditions (see the minimal reproducer in the issue for details).

If scipy 1.15.0 ships OpenBLAS 0.3.28 it might raise the likelihood of triggering this crash. However, running the full scikit-learn test suite with the nightly build of scipy 1.15.0.dev0 that currently ships 0penBLAS 0.3.28 cannot trigger the crash on our CI:

https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=72333&view=logs&j=dfe99b15-50db-5d7b-b1e9-4105c42527cf&t=ef785ae2-496b-5b02-9f0e-07a6c3ab3081&l=40

So maybe the required conditions to trigger the problem are rare enough that this regression would not impact many people in practice.

I reduced the test case to this:

from concurrent.futures import ThreadPoolExecutor
import multiprocessing.pool

import numpy as np

from scipy.linalg import cho_solve


def test_cho_solve_gh_21479():
    from threadpoolctl import threadpool_limits
    threadpool_limits(10, user_api="blas")

    rng = np.random.default_rng(0)
    L = rng.normal(size=(50, 50))
    W_sr = rng.normal(size=50)

    def func(i):
        cho_solve((L, True), np.diag(W_sr))

    with multiprocessing.pool.Pool(2):
        tpe = ThreadPoolExecutor(max_workers=2)
        tpe.map(func, range(10))

The threadpool_limits call is indeed needed - with it the crash happens (thread limit can be 10, or 4 or 20, doesn’t matter), without it the test reliably passes, even with another layer of threads on top with pytest-run-parallel.

I also tested with MKL and Netlib BLAS/LAPACK on Linux, there is no problem there. The problem seems localized to threadpoolctl + OpenBLAS.

I also made it easier to set up a free-threading environment to reproduce problems like this with Pixi:

  1. Ensure you have Pixi installed
  2. git clone https://github.com/rgommers/pixi-dev-scipystack/
  3. cd pixi-dev-scipystack/scipy && git clone https://github.com/rgommers/scipy
  4. cd scipy && git checkout cho-solve-crash (this cho-solve-crash branch)
  5. Run:
$ pixi r build-nogil
$ pixi r test-nogil -t scipy.linalg.tests.test_cho_solve_threading

If you want to use pytest-run-parallel on top:

$ pixi r test-nogil -t scipy.linalg.tests.test_cho_solve_threading -- --parallel-threads=10

If you want to switch from OpenBLAS to MKL or Netlib, edit the pixi.toml in pixi-dev-scipystack/scipy and add the mkl or netlib feature to the free-threading environment and then re-run the pixi r test-nogil ... command. E.g.:

free-threading = {features = ["free-threading", "mkl"], no-default-feature = true }

Thanks for the links @lucascolley, looks like this is one way to install free-threaded CPython from conda-forge:

mamba create -n python-freethreading -c conda-forge/label/python_rc python-freethreading