spack: Installation issue: py-numpy 1.26.1 (builds fine, but can't use) on macOS 12.6.6 with apple-clang@13.1.6 and python 3.11.2 (arm64 platform)
Steps to reproduce the issue
I built py-numpy@1.26.1 on my macOS Monterey (12.6.6; arm64) with apple-clang@13.1.6 and I am running into the buggy Accelerate backend error when importing numpy. It looks like py-numpy is configured correctly to find spack’s openblas for blas and lapack, though:
==> Installing py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l [114/130]
==> No binary for py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/c8/c8c6c72d4a9f831f328efb1312642a1cafafaa88981d9ab76368d50d07d93cbe.tar.gz
==> Applied patch /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/spack/var/spack/repos/builtin/packages/py-numpy/check_executables.patch
==> py-numpy: Executing phase: 'install'
==> [2023-12-15-19:50:08.144175] Find (not recursive): /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.144374] Find complete: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.144742] Find (not recursive): /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/include ['cblas.h', 'cblas.hpp', 'cblas.hxx', 'cblas.hh', 'cblas.H', 'cblas.txx', 'cblas.tcc', 'cblas.icc', 'cblas.mod', 'cblas.inc', 'lapacke.h', 'lapacke.hpp', 'lapacke.hxx', 'lapacke.hh', 'lapacke.H', 'lapacke.txx', 'lapacke.tcc', 'lapacke.icc', 'lapacke.mod', 'lapacke.inc']
==> [2023-12-15-19:50:08.144900] Find complete: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/include ['cblas.h', 'cblas.hpp', 'cblas.hxx', 'cblas.hh', 'cblas.H', 'cblas.txx', 'cblas.tcc', 'cblas.icc', 'cblas.mod', 'cblas.inc', 'lapacke.h', 'lapacke.hpp', 'lapacke.hxx', 'lapacke.hh', 'lapacke.H', 'lapacke.txx', 'lapacke.tcc', 'lapacke.icc', 'lapacke.mod', 'lapacke.inc']
==> [2023-12-15-19:50:08.145243] Find (not recursive): /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.145297] Find complete: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.145631] Find (not recursive): /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/include ['cblas.h', 'cblas.hpp', 'cblas.hxx', 'cblas.hh', 'cblas.H', 'cblas.txx', 'cblas.tcc', 'cblas.icc', 'cblas.mod', 'cblas.inc', 'lapacke.h', 'lapacke.hpp', 'lapacke.hxx', 'lapacke.hh', 'lapacke.H', 'lapacke.txx', 'lapacke.tcc', 'lapacke.icc', 'lapacke.mod', 'lapacke.inc']
==> [2023-12-15-19:50:08.145766] Find complete: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/include ['cblas.h', 'cblas.hpp', 'cblas.hxx', 'cblas.hh', 'cblas.H', 'cblas.txx', 'cblas.tcc', 'cblas.icc', 'cblas.mod', 'cblas.inc', 'lapacke.h', 'lapacke.hpp', 'lapacke.hxx', 'lapacke.hh', 'lapacke.H', 'lapacke.txx', 'lapacke.tcc', 'lapacke.icc', 'lapacke.mod', 'lapacke.inc']
==> [2023-12-15-19:50:08.152210] Find (not recursive): /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.152310] Find complete: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.152668] Find (not recursive): /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.152723] Find complete: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/openblas-0.3.24-zvyufb7/lib ['libopenblas.so', 'libopenblas.dylib', 'openblas.so', 'openblas.dylib']
==> [2023-12-15-19:50:08.155073] '/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/python-3.11.2-u4gyvf4/bin/python3.11' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/py-numpy-1.26.1-zjmtp4y' '--config-settings=builddir=build' '--config-settings=compile-args=-j4' '--config-settings=setup-args=-Dblas=openblas' '--config-settings=setup-args=-Dlapack=openblas' '.'
Using pip 23.1.2 from /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/py-pip-23.1.2-dyzqitm/lib/python3.11/site-packages/pip (python 3.11)
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-build-tracker-8um0elbl
Initialized build tracking at /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-build-tracker-8um0elbl
Created build tracker: /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-build-tracker-8um0elbl
Entered build tracker: /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-build-tracker-8um0elbl
Created temporary directory: /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-install-79ybclvx
Created temporary directory: /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-ephem-wheel-cache-8kc2f7lw
Processing /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src
Added file:///Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src to build tracker '/private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-build-tracker-8um0elbl'
Created temporary directory: /private/var/folders/gb/w8lys0xn3c35rbw_5vfq4mkxxgj2gv/T/pip-modern-metadata-a0eom8qc
Preparing metadata (pyproject.toml): started
Running command Preparing metadata (pyproject.toml)
+ /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/python-3.11.2-u4gyvf4/bin/python3.11 /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src/vendored-meson/meson/meson.py setup /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Dblas=openblas -Dlapack=openblas --native-file=/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src/build/meson-python-native-file.ini
The Meson build system
Version: 1.2.99
Source dir: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src
Build dir: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src/build
Build type: native build
Project name: NumPy
Project version: 1.26.1
C compiler for the host machine: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/spack/lib/spack/env/clang/clang (clang 13.1.6 "Apple clang version 13.1.6 (clang-1316.0.21.2.3)")
C linker for the host machine: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/spack/lib/spack/env/clang/clang ld64 762
C++ compiler for the host machine: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/spack/lib/spack/env/clang/clang++ (clang 13.1.6 "Apple clang version 13.1.6 (clang-1316.0.21.2.3)")
C++ linker for the host machine: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/spack/lib/spack/env/clang/clang++ ld64 762
Cython compiler for the host machine: cython (cython 0.29.36)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python found: YES (/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/python-3.11.2-u4gyvf4/bin/python3.11)
Found pkg-config: /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/unified-env/install/apple-clang/13.1.6/pkg-config-0.29.2-dhli636/bin/pkg-config (0.29.2)
Run-time dependency python found: YES 3.11
Has header "Python.h" with dependency python-3.11: YES
Compiler for C supports arguments -fno-strict-aliasing: YES
Compiler for C supports arguments -ftrapping-math: YES
Message: During parsing cpu-dispatch: The following CPU features were ignored due to platform incompatibility or lack of support:
"XOP FMA4"
Test features "NEON NEON_FP16 NEON_VFPV4 ASIMD" : Supported
Test features "ASIMDHP" : Supported
Test features "ASIMDFHM" : Supported
Configuring npy_cpu_dispatch_config.h using configuration
Message:
CPU Optimization Options
baseline:
Requested : min
Enabled : NEON NEON_FP16 NEON_VFPV4 ASIMD
dispatch:
Requested : max -xop -fma4
Enabled : ASIMDHP ASIMDFHM
Library m found: YES
Found CMake: /opt/homebrew/bin/cmake (3.27.0)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency scipy-openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: YES 0.3.24
Message: BLAS symbol suffix:
Run-time dependency openblas found: YES 0.3.24
Program _build_utils/process_src_template.py found: YES (/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/python-3.11.2-u4gyvf4/bin/python3.11 /Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src/numpy/_build_utils/process_src_template.py)
Program _build_utils/tempita.py found: YES (/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/cache/build_stage/spack-stage-py-numpy-1.26.1-zjmtp4ydxr6fzsbfoz5sjhm6vikf3x5l/spack-src/numpy/_build_utils/tempita.py)
Configuring __config__.py using configuration
Checking for size of "short" : 2
Checking for size of "int" : 4
Checking for size of "long" : 8
Checking for size of "long long" : 8
Checking for size of "float" : 4
Checking for size of "double" : 8
Checking for size of "long double" : 8
Checking for size of "off_t" : 8
...
@adamjstewart @rgommers mentioning you here as the maintainers of the package as per instructions below - have you seen this? I am going to try building on Linux and also on my other macOS (Sonoma 15.x.y, apple-clang@15.n.m) to see if my old macOS version has anything to do with it.
Error message
> which python3
/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/python-3.11.2-u4gyvf4/bin/python3
heinzell@JCSDA-L-18146:~ [brew-arch64]> python3
Python 3.11.2 (main, Dec 15 2023, 08:07:15) [Clang 13.1.6 (clang-1316.0.21.2.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/heinzell/work/spack-stack/spack-stack-add-esmf860/envs/gsi-addon-env/install/apple-clang/13.1.6/py-numpy-1.26.1-zjmtp4y/lib/python3.11/site-packages/numpy/__init__.py", line 404, in <module>
raise RuntimeError(msg)
RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend.
If you compiled yourself, more information is available at:
https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries
Otherwise report this to the vendor that provided NumPy.
RuntimeWarning: invalid value encountered in divide
Information on your system
* **Spack:** 0.21.0 (af1cd513cc1a15ae411e7be3110b8b106fdfaca6)
* **Python:** 3.8.9
* **Platform:** darwin-monterey-m1
* **Concretizer:** clingo
Additional information
spack-build-01-install-out.txt spack-build-env.txt spack-build-out.txt
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have run
spack maintainers <name-of-the-package>and @mentioned any maintainers - I have uploaded the build log and environment files
- I have searched the issues of this repo and believe this is not a duplicate
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (15 by maintainers)
I had promised to wait with updating my mac to test the bug fix. I should have watched the issue on numpy, but I didn’t and therefore didn’t know that it was fixed in numpy until now. And I no longer have this old version of macOS around. Long story short, let’s close this as resolved.
Happy to wait, I have other machines to work on. Appreciate your efforts to fix this!
FWIW, I repeated the build on my other Mac (Sonoma 14.2, apple-clang@15, Intel architecture) and I am not seeing the same build failures (with 1.26.1).
I am still seeing a build failure for py-mpi4py (@3.1.4) on that mac that I am not seeing with python@3.10.8 (our “standard” python that works on both macs fine):
clang: error: invalid argument '-bundle' not allowed with '-dynamiclib'.But the failures reported above are gone. Unfortunately, there are three differences between the setups, the architecture, the compiler version, and the OS version, making it difficult to tell what the problem is.