spack: Builtin make on macOS often does not work with any -j besides -j1 (Permission denied)
Steps to reproduce
This is something that I have noticed on unrelated project: when I use the default /usr/bin/make it has the following behavior:
lfortran/integration_tests(cpp14) $ make -j
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: *** [CMakeFiles/return_04.dir/all] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/sin_03.dir/all] Error 1
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: *** [CMakeFiles/sin_01.dir/all] Error 1
make[1]: *** [CMakeFiles/abs_03.dir/all] Error 1
[ 0%] Building Fortran object CMakeFiles/class_02.dir/class_02.f90.o
[ 0%] Linking Fortran executable class_02
[ 0%] Built target class_02
make: *** [all] Error 2
lfortran/integration_tests(cpp14) $ make -j3
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: *** [CMakeFiles/sin_03.dir/all] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/sin_01.dir/all] Error 1
[ 0%] Building Fortran object CMakeFiles/abs_03.dir/abs_03.f90.o
[ 1%] Linking Fortran executable abs_03
[ 1%] Built target abs_03
make: *** [all] Error 2
lfortran/integration_tests(cpp14) $ make -j2
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: /Library/Developer/CommandLineTools/usr/bin/make: Permission denied
make[1]: *** [CMakeFiles/sin_03.dir/all] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/sin_01.dir/all] Error 1
make: *** [all] Error 2
lfortran/integration_tests(cpp14) $ make -j1
[ 0%] Building Fortran object CMakeFiles/sin_03.dir/sin_03.f90.o
[ 0%] Linking Fortran executable sin_03
[ 0%] Built target sin_03
[ 0%] Building Fortran object CMakeFiles/sin_01.dir/sin_01.f90.o
[ 0%] Linking Fortran executable sin_01
[ 0%] Built target sin_01
[ 1%] Built target abs_03
[ 1%] Built target class_02
[ 1%] Building Fortran object CMakeFiles/return_04.dir/return_04.f90.o
[ 2%] Linking Fortran executable return_04
[ 2%] Built target return_04
[ 2%] Building Fortran object CMakeFiles/return_02.dir/return_02.f90.o
[ 3%] Linking Fortran executable return_02
[ 3%] Built target return_02
[ 3%] Building Fortran object CMakeFiles/return_01.dir/return_01.f90.o
[ 3%] Linking Fortran executable return_01
[ 3%] Built target return_01
...
But if I use the make from Conda, then it works just fine in parallel.
I am reporting this as a standalone issue, because this seems to be happening on both of my two macs, the MacBook Pro 2019 (Intel) and 2020 (M1). As a consequence, this then triggers hard to debug errors, in particular:
They are all caused by the fact that Spack calls the make from macOS with -j8 (or whatever) and it fails with permission issues. However you can’t tell that they are coming from make itself! Very hard to debug.
It’s probably some permission issue, but what could it be? It would be nice to figure out, now when we have isolated the bug.
And a workaround is also super simple: just use spack install -j1. Now, it gets tricky, because due to this Spack also does not bootstrap, but one can just see what packages it is trying to build, and then build them directly with spack install -j1 zlib and such. That way it seems one can build everything.
Error message
N/A
Information on your system
N/A
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have searched the issues of this repo and believe this is not a duplicate
- I have run the failing commands in debug mode and reported the output
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (17 by maintainers)
Poor macOS users
I suspect that too that it is a misconfiguration on my end, although not my personal fault.
But at the end of the day, I’ve seen that over the past 20 years over and over that a system package is broken (at Linux or macOS), one way or another. And the solution is always to simply not depend on it. Whether in Sage, HashDist or now Spack. It’s always the same story and always the same solution.
I don’t think a short-term goal is to have spack bootstrap ab initio :p currently we assume the os provides proper build tools, as well as a working python installation, which usually isn’t too much to ask…
It’s probably easier to bootstrap by downloading working binaries from somewhere (which we currently don’t fully provide except for clingo), and then run
spack install spack, like gcc builds gcc…I love this view. For linux I’ve created https://github.com/eth-cscs/spack-batteries-included which is based on the ideas of AppImage, basically a single-file executable that mounts a zstd-compressed squashfs file included in the binary as an immutable filesystem which contains not only spack, but also
python,make,git,ca-certificates-mozilla, etc. If there’s no fuse on your system, it still contains the tools to extract the squashfs file as if it were a tarball with./spack.x --squashfs-extractand allows you to call./spack/bin/spackdirectly.However, it turns out, even if you go through all this trouble, you will find that build systems themselves include absolute paths to system executables, notably almost all autotools configure scripts pointing to /bin/file 😦 and file can’t always deal with the file database provided by the squashfs file. Also, compilers are still required on the host system.
Also, there is: https://github.com/spack/spack/pull/25979
I would try installing the full XCode suite and launching XCode and signing that agreement. The command-line-tool-only package has given me trouble in the past.