scikit-learn: Unable to successfully install scikit-learn development version with MacBookPro with chip Apple M1 Pro

Describe the bug

I installed the development version of scikit-learn on my computer (MacBookPro with chip Apple M1 Pro). When I ran pytest I got the following error:

(sklearn-dev) ➜  scikit-learn git:(main) pytest
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/maren/Documents/scikit-learn, configfile: setup.cfg, testpaths: sklearn
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collecting ... [1]    54294 killed     pytest

I investigated what’s going on an this is the result: https://gist.github.com/marenwestermann/9ffddb7a2f0ef6798d350f3595997ed1

I’m using a conda environment. I used the Miniforge3-MacOSX-arm64 download from here: https://github.com/conda-forge/miniforge#miniforge

This is the content of my sklearn-dev conda environment: https://gist.github.com/marenwestermann/c01fc4d3044522e329ce83d16680e1a5

Steps/Code to Reproduce

pytest
python -vvv -c "import sklearn"

Expected Results

The expected result would be that when the command pytest is run all test are either skipped or are successful.

Actual Results

Pytest result:

(sklearn-dev) ➜  scikit-learn git:(main) pytest
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 3.9.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/maren/Documents/scikit-learn, configfile: setup.cfg, testpaths: sklearn
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collecting ... [1]    54294 killed     pytest

Further investigation: https://gist.github.com/marenwestermann/9ffddb7a2f0ef6798d350f3595997ed1

Versions

function>

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 29 (29 by maintainers)

Most upvoted comments

From your logs, it looks like the compiler being used is not the conda-forge one. Can you try install the compilers first?

conda create -n sklearn-env -c conda-forge python=3.9 numpy scipy cython compilers
conda activate sklearn-env
mamba install -c conda-forge joblib threadpoolctl pytest
make clean
pip install --verbose --no-build-isolation --editable .

(An alternative is to deactivate and reactivate the environment after installing compilers)

@marenwestermann the upstream bug has been resolved in the conda-forge packages installed via the compilers meta-package. I tried on my machine and confirm the fix. Let me know if it does not work for you (in a new env).

Let me close this issue. Thanks for the report.

Here is the bug report: https://github.com/conda-forge/compilers-feedstock/issues/41

My question is why @thomasjpfan can’t reproduce it then.

Maybe he still had a clang-11 based env activated in his path when he ran the tests. Also sometimes the problem would disappear even in my sklearn-env (maybe because I had additional deps that would require clang-11 in my env?). I am not sure.

No this is not it. I tried again, and this time my memory usage is fine I still get the process killed problem on a new env.

Hum I tried again the instructions of https://github.com/scikit-learn/scikit-learn/issues/22581#issuecomment-1056674469 and now I can reproduce the crash. This is really weird, I am sure it did work (after running the pip install command previously).

Note that:

which clang

cannot always be trusted. Better do:

hash -r  # update the PATH cache in zsh/bash
which clang

or use:

where clang

Also, feel free to use python=3.10 nowadays. It was released 6 months ago.