spack: External package support is generally broken
External package support is currently working in only a minimal fashion.
Let me illustrate this with an extensive example. I’ll then follow up with a proposal for moving forward.
Extensive Example
Related to my issue in #2191, I have several mpi installations on my university’s computing cluster. These I’ve specified in my ~/.spack/packages.yaml file like so (non mpi entries represent my preferred versions and variants for those packages):
packages:
hdf5:
version: [1.8.16]
variants: +mpi +szip
py-h5py:
variants: +mpi
python:
variants: +ucs4
py-cython:
version: [0.25.1]
openmpi:
modules:
openmpi@1.4%gcc: openmpi/1.4-gcc
openmpi@1.4%intel: openmpi/1.4-intel
openmpi@1.6.4%gcc@4.7.1: openmpi/1.6.4-gcc-4.7.1
openmpi@1.6.4%intel@13.1: openmpi/1.6.4-intel-13.1
openmpi@1.6.5%gcc@4.7.1: openmpi/1.6.5-gcc-4.7.1
openmpi@1.6.5%intel@14.0: openmpi/1.6.5-intel-14.0
openmpi@1.8.4%gcc@4.9.2: openmpi/1.8.4-gcc-4.9.2
openmpi@1.8.4%intel@15.0: openmpi/1.8.4-intel-15.0
buildable: False
mpich:
modules:
mpich@3.1.3%gcc@4.7.1: mpi/mpich/3.1.3-gcc-4.7.1
buildable: False
mvapich2:
modules:
mvapich2@1.6%gcc: mvapich2/1.6-gcc
mvapich2@1.6%gcc+debug: mvapich2/1.6-gccdebug
mvapich2@1.6%intel: mvapich2/1.6-intel
mvapich2@1.9b%intel@13.1: mvapich2/1.9b-intel-13.1
mvapich2@2.0b%gcc@4.7.1: mvapich2/2.0b-gcc-4.7.1
mvapich2@2.0b%intel@14.0: mvapich2/2.0b-intel-14.0
mvapich2@2.1rc1%gcc@4.9.2: mvapich2/2.1rc1-gcc-4.9.2
mvapich2@2.1rc1%intel@15.0: mvapich2/2.1rc1-intel-15.0
buildable: False
all:
compiler: [gcc@4.9.2]
providers:
mpi: [mvapich2@2.1rc1]
Now you’ll notice at the end that I’ve specified that the only provider for mpi I want to allow for the moment is mvapich2@2.1rc1. However when I run spack providers I get the following:
spack providers mpi
intel-parallel-studio@cluster:+mpi mpich@1: mpich@3: mvapich2@1.9 mvapich2@2.0: openmpi@1.6.5 openmpi@1.7.5: openmpi@2.0.0:
After looking at the builtin packages, it seems these are all the different builtin packages which provide mpi.
What I would expect however is for the only entry to be mvapich2@2.1rc1.
If spack providers only displayed the results from the builtin repository that would be one thing. However it turns out that this really reflects what providers spack thinks are available. This means that unless you directly specify that a package should depend on one of these external packages spack will fail to realize that it shouldn’t be building it!
For instance, I’m trying to get py-yt running on this cluster. py-yt depends on py-h5py which depends on hdf5. Now I want a specific version of hdf5 with a specific version of mpi which is already on the cluster so being as pedantic as possible I try to get around these problems like so:
- Install
hdf5how I want ithdf5@1.8.16+mpi+szip^mvapich2@2.1rc1. This works. - Get the hash of
hdf5 - Install
py-h5pydepending on that specific hash ofhdf5that I just built.spack install py-h5py^/mzyuler^python@2.7.12. This fails.
(ideally based on my ~/.spack/packages.yaml file this ought to be accomplished just how I expect with spack install hdf5 only)
spack realizes that mvapich2 is part of the dependency tree and ignores the fact that the package is listed as buildable: False in ~/.spack/packages.yaml as well as that it exists as an external package.
Proposal for moving forward
I was poking around the concretizer and provider code to better understand how they work. It seems that packages have a ‘namespace’ which for packages in the builtin repository is builtin. providers for each namespace (builtin is the only namespace as far as I know) then providers get an index which goes here: ~/.spack/cache/providers
My proposal is to add a namespace external which would contain all external packages mentioned in ~/.spack/packages.yaml. These would then be treated as fully installed packages which can participate in the DAG.
I expect that my understanding of spack is lacking. Thus my proposal probably needs some serious tweaks. Please help me hone this proposal into something sensible we can go about building and adding to spack to make dealing with external packages more manageable.
About this issue
- Original URL
- State: closed
- Created 10 years ago
- Comments: 17 (17 by maintainers)
A small correction. When you specify:
You’re not telling Spack that the only provider of mpi is mvapich2@2.1rc1. You’re telling Spack that it should prefer mvapich2@2.1rc1 over other mpi providers when other things are equal. If, for example, a package explicitly asks for mpi@3: and mvapich2@2.1rc1 doesn’t provide that, then Spack might pick openmpi. The likely source of this bug is that Spack decided mvapich2@2.1rc1 didn’t fit in in the DAG, for some reason that will need investigation, and it moved on to the next available MPI.
One possibility is that 2.1rc1 isn’t a known version to Spack’s default mvapich2 package. Have you added that version in your own mvapich2/package.py file? If not that could cause oddities in concretization and may be the source of the bug.