scikit-learn: Segmentation Fault in KMeans on OSX

Could you provide the full output of:

import sklearn
sklearn.show_versions()

You can also quickly try to upgrade to the latest release (1.1.1) to check if the problem is resolved.

conda update scikit-learn -c conda-forge

Thanks @ogrisel and @jjerphan. I can confirm that using conda-forge channel for numpy, scipy and scikit-learn resolves the issue. Should this be explicitly mentioned in the documentation? Currently, the installation instructions say:

If you have not installed NumPy or SciPy yet, you can also install these using conda or pip. When using pip, please ensure that binary wheels are used, and NumPy and SciPy are not recompiled from source, which can happen when using particular configurations of operating system and hardware (such as Linux on a Raspberry Pi).

You could add something like this:

When using conda on MacOS (only MacOS and only on Intel Macs?), NumPy and SciPy must be installed from the same channel as scikit-learn (conda-forge) to avoid compatibility issues between Intel OpenMP and LLVM OpenMP.

Solutions

Here are two options for those facing this issue:

Option 1: Use conda-forge as the only channel (not recommended)

name: test-env
channels:
  - conda-forge
dependencies:
  - python=3.10
  - scikit-learn==1.3.2
 # add other packages as needed

Note that any other packages you add to the environment will have to be available in conga-forge channel, unless explicitly specified otherwise. This is why this solution is probably not recommended for most scenarios.

Option 2: Use conda-forge for scikit-learn and its dependencies (recommended)

name: test-env2
channels:
  - defaults
  - conda-forge
  # add any other channels as needed
dependencies:
  - python=3.10
  - conda-forge::numpy
  - conda-forge::scipy
  - conda-forge::scikit-learn==1.3.2
 # add other packages as needed

Note that, unlike in Option 1, other packages can be installed from different sources you specify under channels (assuming they are all compatible with one another). So this is probably be the recommended solution.

Dependencies

The above solutions produce different dependencies (at least on an Intel Mac), but both have been tested to work with Kmeans as expected. Below is the output of sklearn.show_versions().

Option 1

System:
    python: 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:09:20) [Clang 16.0.6 ]
executable: /Users/alinacherkas/anaconda3/envs/test-env/bin/python
   machine: macOS-14.1-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.3.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.0
        scipy: 1.11.3
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/alinacherkas/anaconda3/envs/test-env/lib/libopenblasp-r0.3.24.dylib
        version: 0.3.24
threading_layer: openmp
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/alinacherkas/anaconda3/envs/test-env/lib/libomp.dylib
        version: None

Option 2

System:
    python: 3.10.13 (main, Sep 11 2023, 08:39:02) [Clang 14.0.6 ]
executable: /Users/alinacherkas/anaconda3/envs/test-env2/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.3.2
          pip: 23.3
   setuptools: 68.0.0
        numpy: 1.26.0
        scipy: 1.11.3
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: /Users/alinacherkas/anaconda3/envs/test-env2/lib/libopenblasp-r0.3.21.dylib
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.21
    num_threads: 4
threading_layer: pthreads
   architecture: Haswell

       filepath: /Users/alinacherkas/anaconda3/envs/test-env2/lib/libomp.dylib
         prefix: libomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8

Thanks for the report. Indeed, this configuration is known to crash. The llvm libomp and the intel libiomp are known to cause a crash when installed together: either install all numpy, scipy, scikit-learn from the defaults channel (and you should only get libiomp) or install everything from the conda-forge channel (and you should get libomp alone).

@matthewnour and others who can reproduce the problem, could you please post the output on sklearn.show_versions() in the environment that can trigger the problem? See @jacktang’s report here.

https://github.com/scikit-learn/scikit-learn/issues/23574#issuecomment-1248889181

In particular I am interested if you also have torch’s openmp in the output and if so how did you install torch on your machine: do you use pip or conda. Ideally please include a set of pip or conda commands that make it possible to create a new venv or conda env that can trigger the problem.

Personally I do not experience this problem on my M1 laptop and I install most dependencies from conda-forge (using mambaforge): https://github.com/conda-forge/miniforge#mambaforge

Same issue for me. M1 Mac. Incase anyone has found a solution

@jjerphan , yes, occasionally I use other conda channels for packages that instruct so. For example, plotly::plotly, plotly::python-kaleido and pytorch::pytorch.

You could add something like this:

When using conda on MacOS (only MacOS and only on Intel Macs?), NumPy and SciPy must be installed from the same channel as scikit-learn (conda-forge) to avoid compatibility issues between Intel OpenMP and LLVM OpenMP.

I think adding a short sentence to the install docs would be good. I’d make it something like the advice from https://github.com/scikit-learn/scikit-learn/issues/23574#issuecomment-1782261722 - “make sure to install numpy, scipy and scikit-learn from the same conda channel”.

People in the situation described by @ogrisel might be interested in reading those notes from threadpoolctl.

Also note, we just merged #27167 that might or might not impact the crash you observed in this issue. If anybody who could reproduce #23574 can tell me if #27167 fixes the problem for them, that would be very nice.

It would be very helpful if you could:

• post the output of conda list for future reference
• try if the problem persists in a new conda environment using the conda-forge channel, conda create -n test-env -c conda-forge scikit-learn
• try to use a gdb or ldb to get a stack-trace from the segmentation fault, see https://github.com/scikit-learn/scikit-learn/issues/23574#issuecomment-1152379936 for gdb

@EoinKenny as a work-around you may want to try to create a new environment from conda-forge and see whether the problem persists, i.e. something like this

conda create -c conda-forge -n test-env scikit-learn -y
conda activate test-env
python debugging.py

You could also use gdb to see whether it gives some useful additional info

gdb run --args python debugging.py

Then type r (for run) and then bt (for backtrace). There maybe more info about gdb and python in https://wiki.python.org/moin/DebuggingWithGdb.