spack: macOS M1: Spack fails to bootstrap without Xcode
I started from a clean repository updated to the latest develop branch (c2e1a12cdf44edd7f29e465bbdfdc511b97d9669).
$ spack compilers
==> Available compilers
-- apple-clang bigsur-arm64 -------------------------------------
apple-clang@12.0.0
~ $ spack spec qemu
Input spec
--------------------------------
qemu
Concretized
--------------------------------
==> Bootstrapping clingo from pre-built binaries
==> Bootstrapping clingo from sources
==> Installing gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> No binary for gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/69/69b6d2868e70167ba1bdb9030b49beeb20f00b37e30825e83fd04291d96bc5f7.tar.gz
==> No patches needed for gnuconfig
==> gnuconfig: Executing phase: 'install'
==> gnuconfig: Successfully installed gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
Fetch: 0.85s. Build: 0.19s. Total: 1.03s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/gnuconfig-2021-08-14-gedovjnheiii5r4na5e3pjlekcergep6
==> Installing zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr
==> No binary for zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/c3/c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr
Fetch: 1.06s. Build: 1.68s. Total: 2.74s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/zlib-1.2.11-a3p23rnymc7d6ens3mk7wj75r5sucdvr
[+] /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8 (external python-3.8-yumcagn3zdktmhjepxgxvbokqdl24zaz)
==> Installing libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk
==> No binary for libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/e6/e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> libiconv: Successfully installed libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk
Fetch: 2.18s. Build: 18.15s. Total: 20.34s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/libiconv-1.16-fyaei4lfz3n7nzpna2vetg4a5uq7pdbk
==> Installing libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc
==> No binary for libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/be/be78ee4176b05f7c75ff03298d84874db90f4b6c9d5503f0da1226b3a3c48119.tar.gz
==> No patches needed for libsigsegv
==> libsigsegv: Executing phase: 'autoreconf'
==> libsigsegv: Executing phase: 'configure'
==> libsigsegv: Executing phase: 'build'
==> libsigsegv: Executing phase: 'install'
==> libsigsegv: Successfully installed libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc
Fetch: 0.96s. Build: 4.42s. Total: 5.38s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/libsigsegv-2.13-jx46dmox6wtqotnsvvcrqwx6jfuj72pc
==> Installing berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea
==> No binary for berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0c/0cecb2ef0c67b166de93732769abdeba0555086d51de1090df325e18ee8da9c8.tar.gz
==> Applied patch /Users/ondrej/repos/spack/var/spack/repos/builtin/packages/berkeley-db/drop-docs.patch
==> Ran patch() for berkeley-db
==> berkeley-db: Executing phase: 'autoreconf'
==> berkeley-db: Executing phase: 'configure'
==> berkeley-db: Executing phase: 'build'
==> berkeley-db: Executing phase: 'install'
==> berkeley-db: Successfully installed berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea
Fetch: 9.03s. Build: 26.26s. Total: 35.30s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/berkeley-db-18.1.40-7q4rqdl4vtulxj4jx56l2mac2d2embea
==> Installing re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau
==> No binary for re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0f/0fc45e4130a8a555d68e230d1795de0216dfe99096b61b28e67c86dfd7d86bda.tar.xz
==> No patches needed for re2c
==> re2c: Executing phase: 'autoreconf'
==> re2c: Executing phase: 'configure'
==> re2c: Executing phase: 'build'
==> re2c: Executing phase: 'install'
==> re2c: Successfully installed re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau
Fetch: 1.38s. Build: 20.24s. Total: 21.62s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/re2c-2.2-ahy7ntwn3wjqkphsppo2cixexk62gmau
==> Installing pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc
==> No binary for pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ef/ef9c7e61822b7cb8356e6e9e1dca58d9556f3200d78acab35e4347e9d4c2bbaf.tar.xz
==> No patches needed for pkgconf
==> pkgconf: Executing phase: 'autoreconf'
==> pkgconf: Executing phase: 'configure'
==> pkgconf: Executing phase: 'build'
==> pkgconf: Executing phase: 'install'
==> pkgconf: Successfully installed pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc
Fetch: 1.07s. Build: 5.01s. Total: 6.08s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/pkgconf-1.8.0-crd2i745cj5p6am2m5sajcycuvr3jonc
==> Installing diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27
==> No binary for diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/a6/a6bdd7d1b31266d11c4f4de6c1b748d4607ab0231af5188fc2533d0ae2438fec.tar.xz
==> No patches needed for diffutils
==> diffutils: Executing phase: 'autoreconf'
==> diffutils: Executing phase: 'configure'
==> diffutils: Executing phase: 'build'
==> diffutils: Executing phase: 'install'
==> diffutils: Successfully installed diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27
Fetch: 1.41s. Build: 32.46s. Total: 33.87s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/diffutils-3.8-tps25qichnyhdskh2yoimpoeersmou27
==> Installing m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a
==> No binary for m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3b/3be4a26d825ffdfda52a56fc43246456989a3630093cced3fbddf4771ee58a70.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/bf/bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89
==> Fetching https://mirror.spack.io/_source-cache/archive/9d/9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573
==> Applied patch http://git.savannah.gnu.org/cgit/m4.git/patch/?id=a1354086
==> Applied patch http://git.savannah.gnu.org/cgit/m4.git/patch/?id=cd7f4d15
==> Ran patch() for m4
==> m4: Executing phase: 'autoreconf'
==> m4: Executing phase: 'configure'
==> m4: Executing phase: 'build'
==> m4: Executing phase: 'install'
==> m4: Successfully installed m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a
Fetch: 1.88s. Build: 47.45s. Total: 49.33s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/m4-1.4.19-ghuewuay4dqfxf35fzbqknakd5juhl7a
==> Installing ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p
==> No binary for ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/30/30306e0c76e0f9f1f0de987cf1c82a5c21e1ce6568b9227f7da5b71cbea86c9d.tar.gz
==> No patches needed for ncurses
==> ncurses: Executing phase: 'autoreconf'
==> ncurses: Executing phase: 'configure'
==> ncurses: Executing phase: 'build'
==> ncurses: Executing phase: 'install'
==> ncurses: Successfully installed ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p
Fetch: 2.19s. Build: 55.02s. Total: 57.21s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/ncurses-6.2-exubvurzson2t7xztjvlg7gayp65kg7p
==> Installing bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro
==> No binary for bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ab/ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269.tar.gz
==> Ran patch() for bzip2
==> bzip2: Executing phase: 'install'
==> bzip2: Successfully installed bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro
Fetch: 1.22s. Build: 1.03s. Total: 2.25s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/bzip2-1.0.8-xoi7soo72qnpkswxbdby3a4kmip2e3ro
==> Installing readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig
==> No binary for readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/f8/f8ceb4ee131e3232226a17f51b164afc46cd0b9e6cef344be87c65962cb82b02.tar.gz
==> Ran patch() for readline
==> readline: Executing phase: 'autoreconf'
==> readline: Executing phase: 'configure'
==> readline: Executing phase: 'build'
==> readline: Executing phase: 'install'
==> readline: Successfully installed readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig
Fetch: 1.89s. Build: 8.31s. Total: 10.20s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/readline-8.1-odmkrknw6jd6bu2rdomd5ivloae6gwig
==> Installing gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw
==> No binary for gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/37/37ed12214122b972e18a0d94995039e57748191939ef74115b1d41d8811364bc.tar.gz
==> No patches needed for gdbm
==> gdbm: Executing phase: 'autoreconf'
==> gdbm: Executing phase: 'configure'
==> gdbm: Executing phase: 'build'
==> gdbm: Executing phase: 'install'
==> gdbm: Successfully installed gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw
Fetch: 0.95s. Build: 8.24s. Total: 9.19s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/gdbm-1.19-tcvwmvv2iq7slyhayv5wswqpt7oy3ynw
==> Installing perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
==> No binary for perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w found: installing from source
==> Warning: Failed to detect macOS command line tools version: No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.
==> Fetching https://mirror.spack.io/_source-cache/archive/55/551efc818b968b05216024fb0b727ef2ad4c100f8cb6b43fab615fa78ae5be9a.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/9d/9da50e155df72bce55cb69f51f1dbb4b62d23740fb99f6178bb27f22ebdf8a46.tar.gz
==> Moving resource stage
source : /var/folders/wm/h8g03zb53_sgm6r8r8h9pmpw0000gp/T/ondrej/spack-stage/resource-cpanm-xzib42oxhft2bgakyhhovjy5o3lvip4w/spack-src/
destination : /var/folders/wm/h8g03zb53_sgm6r8r8h9pmpw0000gp/T/ondrej/spack-stage/spack-stage-perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w/spack-src/cpanm/cpanm
==> Ran patch() for perl
==> perl: Executing phase: 'configure'
==> perl: Executing phase: 'build'
==> perl: Executing phase: 'install'
==> perl: Successfully installed perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
Fetch: 6.44s. Build: 1m 30.54s. Total: 1m 36.98s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/perl-5.34.0-xzib42oxhft2bgakyhhovjy5o3lvip4w
==> Installing bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
==> No binary for bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/06/06c9e13bdf7eb24d4ceb6b59205a4f67c2c7e7213119644430fe82fbd14a0abb.tar.gz
==> No patches needed for bison
==> bison: Executing phase: 'autoreconf'
==> bison: Executing phase: 'configure'
==> bison: Executing phase: 'build'
==> bison: Executing phase: 'install'
==> bison: Successfully installed bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
Fetch: 2.27s. Build: 34.04s. Total: 36.31s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/bison-3.8.2-comfky2x72qyu32uakmlltap4sc5jfit
==> Installing openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv
==> No binary for openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0b/0b7a3e5e59c34827fe0c3a74b7ec8baef302b98fa80088d7f9153aa16fa76bd1.tar.gz
==> Ran patch() for openssl
==> openssl: Executing phase: 'install'
==> openssl: Successfully installed openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv
Fetch: 3.51s. Build: 24.16s. Total: 27.67s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/openssl-1.1.1l-xq4viuhggqsoqii6osjf5k25ugjizejv
==> Installing cmake-3.22.1-i4ydo5hgu5g5dzybpmqznbihusnz7hwq
==> No binary for cmake-3.22.1-i4ydo5hgu5g5dzybpmqznbihusnz7hwq found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0e/0e998229549d7b3f368703d20e248e7ee1f853910d42704aa87918c213ea82c0.tar.gz
==> No patches needed for cmake
==> cmake: Executing phase: 'bootstrap'
==> cmake: Executing phase: 'build'
==> cmake: Executing phase: 'install'
==> cmake: Successfully installed cmake-3.22.1-i4ydo5hgu5g5dzybpmqznbihusnz7hwq
Fetch: 3.90s. Build: 3m 22.81s. Total: 3m 26.71s.
[+] /Users/ondrej/.spack/bootstrap/store/darwin-bigsur-aarch64/apple-clang-12.0.5/cmake-3.22.1-i4ydo5hgu5g5dzybpmqznbihusnz7hwq
==> Installing clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc
==> No binary for clingo-bootstrap-spack-7ctouvxrqj6vs2mw2txgl5jfsqfrnpuc found: installing from source
==> Error: RuntimeError: Unable to locate Python3.framework/Versions/3.8/Python3 libraries in /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib
/Users/ondrej/repos/spack/var/spack/repos/builtin/packages/python/package.py:828, in libs:
825 return LibraryList(os.path.join(macos_developerdir, ldlibrary))
826 else:
827 msg = 'Unable to locate {0} libraries in {1}'
>> 828 raise RuntimeError(msg.format(ldlibrary, libdir))
829 else:
830 library = self.config_vars['LIBRARY']
831
==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %apple-clang target=aarch64" due to the following failures:
'spack-install' raised InstallError: Terminating after first install failure: RuntimeError: Unable to locate Python3.framework/Versions/3.8/Python3 libraries in /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib
Please run `spack -d spec zlib` for more verbose error messages
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 22 (22 by maintainers)
My understanding is that the only dependency on Xcode is via depending on Python from it. But it seems that Spack works with any Python package, such as the one from Conda. There is Python installed using commandline utilities, but the paths must be adjusted in Spack for that. Finally, Spack insists on creating some directories in the Python installation, which I don’t quite understand why, and that fails too obviously. But those are all technical issues that are not related to Xcode in my opinion.
I see. Thanks for the feedback. The reason for me to use Conda and Spack on macOS is to bypass the default way of installing packages, which I do not find user friendly or maintainable. Installing Xcode is one of those big packages that I do not like installing. I just like to take the stock macOS as it comes, and just do minimal changes to it. The command line utilities are needed (it seems), so I am fine with that.
I tried to look what it takes to install Xcode, and it is asking me to sign with my Apple ID. I don’t want to do that either.
The good news is that Conda works, and apparently it is able to create the initial environment for Spack to operate in, so that’s fine and I’ll just use Conda instead of Xcode. I do think Spack itself should not require Xcode — I got it working with an earlier version of Spack, the latest version seems even more depending on Xcode, but I don’t think there is any technical reason (necessity) for that.
I mean: installing the 12.7 GB Xcode just to have the python framework in the expected location is a hefty price to pay.
Not sure if/how we can lift the requirement, but as far as I remember we always had this documented as:
at this page
I don’t think my conda installation is interfering with Spack. It doesn’t alter any paths unless I do “conda activate something”.
The apple developer tools documentation:
So I guess the distinction may be that the CLT are for command-line-focused (no GUI components) development and Xcode proper is for development that could involve the gui or other Apple platforms.
Interesting, well the next time I set up a computer I’ll try not installing xcode and see what errors crop up. I think the assumption is generally that if you’re “developing” (i.e. building code) on mac then Xcode is needed — homebrew and conda can likely get away with this because they distribute prebuilt binaries. Perhaps Spack can get to that point as well.
Yes, I am developing on macOS. The CommandLineTools tools have the compiler (clang). I haven’t done GUI, I just develop C++ and Fortran applications, and they all work with just CommandLineTools. I guess I like to keep my system minimal. So far Spack is the first project that requires it.
@certik I found it much easier to bootstrap with homebrew:
brew install gcc,spack compiler find gccOn an M1 Max, I still get exactly the same error as above, but unfortunately the patch above does not fix it, I get further errors. The following patch gets it to start compiling
clingo-bootstrap-spack-fwqxswj4iz5tyic2sjguthrzmundqq3z:But it fails with:
I don’t now how to fix that.
Spack needs to build its own Python, it should not rely on system Python like this. Also note that the lines
mkdirp(module.python_platlib)were trying to create a directory in/Library/...which is unacceptable. Spack should not modify any system files, it should install everything locally without root access.Hmm I’m guessing we need to build bootstrap binaries for
darwin-arm64. The missing python framework may be a separate issue.This error can be fixed by:
As such, I suspect that Spack requires Xcode, while I only have the command line utilities installed. Honestly, things should just work with the command line utilities, as everything else seems to work for me that way.