scikit-learn: V1.5 randomly hangs on import

Describe the bug

Since upgrading to version 1.5.0 we are seeing issues where the process stalls on import. Through a core dump, it appears to be locking up here. We run hundreds of processes in parallel and it will hang on only a handful (less than 10).

Steps/Code to Reproduce

import sklearn

Expected Results

it imports

Actual Results

process hangs indefinitely.

Versions

System:
    python: 3.11.3 (main, May 15 2023, 15:45:52) [GCC 11.2.0]
executable: /home/jcoder/git/neo/neo/pyenv/bin/python
   machine: Linux-6.1.87-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.0
          pip: 24.0
   setuptools: 70.0.0
        numpy: 1.24.4
        scipy: 1.11.1
       Cython: 0.29.35
       pandas: 2.0.2
   matplotlib: 3.7.2
       joblib: 1.3.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /opt/anaconda/2023/envs/pandas2/lib/libopenblasp-r0.3.23.so
        version: 0.3.23
threading_layer: pthreads
   architecture: Zen
    num_threads: 64

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/~~~~~~/lib/python3.11/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 64

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Hi @jeremiedbb I did a pip install git+https://github.com/jeremiedbb/scikit-learn@lazy-threadpool-controller and then ran the tests I have been using to troubleshoot this and everything worked. When running with v1.5.0 the same tests hung on 14 of the 600 processes so in my case, this seems to fix it.