scikit-learn: V1.5 randomly hangs on import
Describe the bug
Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up here. We run hundreds of processes in parallel and it will hang on only a handful (less than 10).
Steps/Code to Reproduce
import sklearn
Expected Results
it imports
Actual Results
process hangs indefinitely.
Versions
System:
python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
machine: Linux-6.1.87-x86_64-with-glibc2.31
Python dependencies:
sklearn: 1.5.0
pip: 24.0
setuptools: 70.0.0
numpy: 1.24.4
scipy: 1.11.1
Cython: 0.29.35
pandas: 2.0.2
matplotlib: 3.7.2
joblib: 1.3.0
threadpoolctl: 3.1.0
Built with OpenMP: True
threadpoolctl info:
user_api: blas
internal_api: openblas
prefix: libopenblas
filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
version: 0.3.23
threading_layer: pthreads
architecture: Zen
num_threads: 64
user_api: openmp
internal_api: openmp
prefix: libgomp
filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
version: None
num_threads: 64
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (9 by maintainers)
Hi @jeremiedbb I did a
pip install git+https://github.com/jeremiedbb/scikit-learn@lazy-threadpool-controllerand then ran the tests I have been using to troubleshoot this and everything worked. When running with v1.5.0 the same tests hung on 14 of the 600 processes so in my case, this seems to fix it.