spack: Running boostrap on an NFSv4 volume produces a PermissionError

Steps to reproduce

Running spack spec zlib on a system where the home directory is on a NAS (nfs v4) volume results in a permission error

==> [2021-09-15-20:10:32.222539] [BOOTSTRAP MODULE clingo] Try installing "clingo-bootstrap" from binary cache at "https://mirror.spack.io/bootstrap/github-actions/v0.1"
==> [2021-09-15-20:10:39.201450] PermissionError: [Errno 1] Operation not permitted: '/nashome/g/gartung/.spack/bootstrap/store/linux-scientific7-piledriver'

Running export HOME=/path/on/local/disk; source spack/share/spack/setup_env.sh;spack spec zlib results in a checksum error. Note: spack clean -a might have also been run.

==> [2021-09-15-20:00:41.427673] [BOOTSTRAP MODULE clingo] Try installing "clingo-bootstrap" from binary cache at "https://mirror.spack.io/bootstrap/github-actions/v0.1"
==> [2021-09-15-20:00:41.902640] NoChecksumException: cannot verify checksum for "/tmp/gartung/spack-stage/build_cache/linux-rhel5-x86_64-gcc-9.3.0-clingo-bootstrap-spack-vcipwnf57slgoo7busvvkzjkk7vydeb5.spack" [expected=db5222760045f20ad1e5c194179d31273b8e4bfa6ade38e15cd3182d685cc05b]

==> [2021-09-15-20:01:25.613745] [BOOTSTRAP MODULE clingo] Unexpected error "Conflicts in concretized spec "clingo-bootstrap@spack%gcc@4.8.5~docs~ipo+python build_type=Release arch=linux-scientific7-x86_64/3sc2o6n"

    List of matching conflicts for spec:

    clingo-bootstrap@spack%gcc@4.8.5~docs~ipo+python build_type=Release arch=linux-scientific7-x86_64
 
1. "%gcc@:5.99.99" conflicts with "clingo-bootstrap" [C++14 support is required to bootstrap clingo]
"

Error message

No response

Information on your system

Running on Scientific Linux 7 a RHEL 7 recompile with system compiler gcc@4.8.5

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 25 (25 by maintainers)

Most upvoted comments

It turns out the problem is the umask 0022 which adds the group sticky bit when the archive for clingo is unpacked in a temporary directory. The buildcache procedure creates another temporary tarfile which preserves the sticky bit. When this temporary tarfile is unpacked on nfsv4 the stick bit makes it impossible for python to write the files into the directories. I think this is because the default group is username.username on /tmp but username.nobody on nfs v4.

For example this is the tempdir where the archive is extracted and repackaged. The extracted directory has the sitcky bit for other because of the umask and the newly created archive of the same name preserves the sticky bit.

-bash-4.2$ pwd
/tmp/tmpx727_e39
-bash-4.2$ ls -l
total 34460
drwxr-sr-x 6 gartung g163     4096 Jun 17 14:25 clingo-bootstrap-spack-vcipwnf57slgoo7busvvkzjkk7vydeb5
-rw-r--r-- 1 gartung g163    10912 Jun 17 14:39 linux-rhel5-x86_64-gcc-9.3.0-clingo-bootstrap-spack-vcipwnf57slgoo7busvvkzjkk7vydeb5.spec.yaml
-rw-r--r-- 1 gartung g163 27187200 Sep 16 09:21 linux-rhel5-x86_64-gcc-9.3.0-clingo-bootstrap-spack-vcipwnf57slgoo7busvvkzjkk7vydeb5.tar
-rw-r--r-- 1 gartung g163  8081379 Jun 17 14:39 linux-rhel5-x86_64-gcc-9.3.0-clingo-bootstrap-spack-vcipwnf57slgoo7busvvkzjkk7vydeb5.tar.gz
tar tfv linux-rhel5-x86_64-gcc-9.3.0-clingo-bootstrap-spack-vcipwnf57slgoo7busvvkzjkk7vydeb5.tar
drwxr-sr-x gartung/g163      0 2021-06-17 14:25 ./
drwxr-sr-x gartung/g163      0 2021-06-17 14:25 ./include/
-rw-r--r-- gartung/g163 226750 2021-06-17 14:07 ./include/clingo.hh
-rw-r--r-- gartung/g163 192647 2021-06-17 14:07 ./include/clingo.h
drwxr-sr-x gartung/g163      0 2021-06-17 14:25 ./lib64/
-rwxr-xr-x gartung/g163 15712417 2021-06-17 14:25 ./lib64/libclingo.so.4.0
drwxr-sr-x gartung/g163        0 2021-06-17 14:25 ./lib64/cmake/
drwxr-sr-x gartung/g163        0 2021-06-17 14:25 ./lib64/cmake/Clingo/
-rw-r--r-- gartung/g163     1026 2021-06-17 14:21 ./lib64/cmake/Clingo/ClingoConfig.cmake
-rw-r--r-- gartung/g163     2742 2021-06-17 14:21 ./lib64/cmake/Clingo/ClingoConfigVersion.cmake
-rw-r--r-- gartung/g163      813 2021-06-17 14:21 ./lib64/cmake/Clingo/ClingoTargets-release.cmake
-rw-r--r-- gartung/g163     3238 2021-06-17 14:21 ./lib64/cmake/Clingo/ClingoTargets.cmake
lrwxrwxrwx gartung/g163        0 2021-09-16 09:21 ./lib64/libclingo.so -> libclingo.so.4
lrwxrwxrwx gartung/g163        0 2021-09-16 09:21 ./lib64/libclingo.so.4 -> libclingo.so.4.0

I am debugging the buildcache install command. I suspect something in it is causing the permission error on nfsv4.

/tmp is local disk.