spack: Bootstrap issue: cannot find Python (new apple M2)

Steps to reproduce the issue

$ spack spec zlib

@alalazo I’m setting up a brand-new M2 macbook air. I’ve installed XCode and CLT (and used xcode-select --reset to cause the default tools to point back to xcode). The python command isn’t available but python3 is: Python 3.8 exists at /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8 , but the path Spack passes (/Library/Python/3.8/include) does not: there is no /Library/Python.

Error message

Error message
==> Installing clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4
==> No binary for clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4 found: installing from source
==> Using cached archive: /opt/spack/var/spack/cache/_source-cache/git//potassco/clingo.git/2a025667090d71b2c9dce60fe924feb6bde8f667.tar.gz
==> Warning: Fetching from mirror without a checksum!
  This package is normally checked out from a version control system, but it has been archived on a spack mirror.  This means we cannot know a checksum for the tarball in advance. Be sure that your connection to this mirror is secure!
==> Already patched clingo-bootstrap
==> clingo-bootstrap: Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    'cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_FIND_FRAMEWORK:STRING=LAST' '-DCMAKE_FIND_APPBUNDLE:STRING=LAST' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/lib;/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/lib64;/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib' '-DCMAKE_PREFIX_PATH:STRING=/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/re2c-2.2-ax6z5gjajrwybsvlnktzrohv2qqhkzbn;/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.2-syy3udkw2khnxsvhaa6wni3oi3q4ezuj;/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/bison-3.8.2-m6v6mtbqsy7ws5ydetr7za6fewxnhw5l;/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8' '-DCLINGO_BUILD_WITH_LUA=OFF' '-DCLINGO_REQUIRE_PYTHON=ON' '-DCLINGO_BUILD_WITH_PYTHON=ON' '-DPYCLINGO_USER_INSTALL=OFF' '-DPYCLINGO_USE_INSTALL_PREFIX=ON' '-DCLINGO_BUILD_PY_SHARED:STRING=OFF' '-DPython_EXECUTABLE:STRING=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8' '-DPython_INCLUDE_DIR:STRING=/Library/Python/3.8/include' '-DCLINGO_BUILD_APPS:STRING=OFF' '/var/folders/n9/mqnx20b929z469f6p3fbq7c40000gn/T/seth/spack-stage/spack-stage-clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/spack-src'

1 error found in build log:

3 CMake Error at /Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmak e-3.23.2-syy3udkw2khnxsvhaa6wni3oi3q4ezuj/share/cmake-3.23/Modules/FindPackageHandleStandardArgs. cmake:230 (message): 4 Could NOT find Python (missing: Python_LIBRARIES Interpreter Development 5 Development.Module Development.Embed) 6 Call Stack (most recent call first): 7 /Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.2-syy3 udkw2khnxsvhaa6wni3oi3q4ezuj/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_F PHSA_FAILURE_MESSAGE) 8 /Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.2-syy3 udkw2khnxsvhaa6wni3oi3q4ezuj/share/cmake-3.23/Modules/FindPython.cmake:561 (find_package_handle_s tandard_args) 9 CMakeLists.txt:130 (find_package)

See build log for details: /var/folders/n9/mqnx20b929z469f6p3fbq7c40000gn/T/seth/spack-stage/spack-stage-clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/spack-build-out.txt ==> Error: cannot bootstrap the “clingo” Python module from spec “clingo-bootstrap@spack+python %apple-clang target=aarch64” due to the following failures: github-actions-v0.1 raised ValueError: source is not trusted spack-install raised InstallError: Terminating after first install failure: ProcessError: Command exited with status 1: ‘cmake’ ‘-G’ ‘Unix Makefiles’ ‘-DCMAKE_INSTALL_PREFIX:STRING=/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4’ ‘-DCMAKE_BUILD_TYPE:STRING=Release’ ‘-DBUILD_TESTING:BOOL=OFF’ ‘-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF’ ‘-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON’ ‘-DCMAKE_FIND_FRAMEWORK:STRING=LAST’ ‘-DCMAKE_FIND_APPBUNDLE:STRING=LAST’ ‘-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON’ ‘-DCMAKE_INSTALL_RPATH:STRING=/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/lib;/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/lib64;/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib’ ‘-DCMAKE_PREFIX_PATH:STRING=/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/re2c-2.2-ax6z5gjajrwybsvlnktzrohv2qqhkzbn;/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/cmake-3.23.2-syy3udkw2khnxsvhaa6wni3oi3q4ezuj;/Users/seth/.spack/bootstrap/store/darwin-monterey-aarch64/apple-clang-13.1.6/bison-3.8.2-m6v6mtbqsy7ws5ydetr7za6fewxnhw5l;/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8’ ‘-DCLINGO_BUILD_WITH_LUA=OFF’ ‘-DCLINGO_REQUIRE_PYTHON=ON’ ‘-DCLINGO_BUILD_WITH_PYTHON=ON’ ‘-DPYCLINGO_USER_INSTALL=OFF’ ‘-DPYCLINGO_USE_INSTALL_PREFIX=ON’ ‘-DCLINGO_BUILD_PY_SHARED:STRING=OFF’ ‘-DPython_EXECUTABLE:STRING=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8’ ‘-DPython_INCLUDE_DIR:STRING=/Library/Python/3.8/include’ ‘-DCLINGO_BUILD_APPS:STRING=OFF’ ‘/var/folders/n9/mqnx20b929z469f6p3fbq7c40000gn/T/seth/spack-stage/spack-stage-clingo-bootstrap-spack-bunntglo2egkmjanvwkhqwejq5ydonh4/spack-src’ Run spack --debug ... for more detailed errors

Information on your system

  • Spack: 0.19.0.dev0 (babb0edf077dfccf8e66ab63784437b06bdc3963)
  • Python: 3.8.9
  • Platform: darwin-monterey-m1
  • Concretizer: clingo

Additional information

spack-build-out.txt spack-build-env.txt

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
  • I have uploaded the build log and environment files
  • I have searched the issues of this repo and believe this is not a duplicate

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 23 (23 by maintainers)

Most upvoted comments

I think the solution is to look at the INCLUDEPY setting. CONFINCLUDEPY also contains the correct directory. We have a ton of methods to search for libraries but no one encountered issues with headers until now. I’ll submit a PR to fix spec['python'].headers.

success! I deleted clingo-bootstrap and completed the boostrapping with system python. 🎉

And yes, there is a true python include dir, at least if xcode is installed, at /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8. CMake seems to detect this automatically.

OK hold on: I think that last failure might’ve been due to a stale CMakeCache that the boostrap reused. A standalone build with the env and the updated commands allowed clingo-bootstrap to configure. I’m continuing to try.

Yes, just removing the include dir works. Removing it when the directory doesn’t exist definitely sounds like the quickest path to a successful bootstrap.

This is the monolithic PR where I attempted to fix this, or at least get us most of the way there: https://github.com/spack/spack/pull/30834

See this note in the Python package for a bunch of good references:

    # https://docs.python.org/3/library/sysconfig.html#installation-paths                
    # https://discuss.python.org/t/understanding-site-packages-directories/12959         
    # https://github.com/pypa/pip/blob/22.1/src/pip/_internal/locations/__init__.py      
    # https://github.com/pypa/installer/pull/103                                         
                                                                                         
    # NOTE: XCode Python's sysconfing module was incorrectly patched, and hard-codes     
    # everything to be installed in /Library/Python. Therefore, we need to use a         
    # fallback in the following methods. For more information, see:                      
    # https://github.com/pypa/pip/blob/22.1/src/pip/_internal/locations/__init__.py#L486 

That’s being set in the cmake_python_hints property in the clingo package. We query the python package and ask it where it’s headers live. The problem is that the system Python uses a non-standard layout but fails to patch sysconfig so sysconfig has no idea where the header files actually live. I added a fallback to <prefix>/include but if the actual location is different then that won’t work either. If you remove the -DPython_INCLUDE_DIR do things work? Maybe we should just always remove this. At least when the directory doesn’t exist.

FindPython is something maintained by Kitware (but open to contributions). I’ve submitted patches to mainline CMake before so I’ll see if I can take a crack at it.

That did the trick, thanks @adamjstewart ! Maybe we need to open a CMake issue to fix system python (especially if CLT or Xcode are installed).

Try unsetting SPACK_PYTHON or explicitly setting it to the right exe. Spack caches this and it controls which exe is used to run Spack (and therefore to bootstrap)

Also, it’s interesting that your spec says M1 instead of M2. @alalazo is that something that needs to be fixed in archspec?

CMake doesn’t know how to find the system Python on macOS. Until we have pre-compiled clingo binaries for macOS arm64, you’ll need to first install Python (homebrew, conda, source, etc.).