spack: spack load: don't load multiple variants of same packages and don't load packages that depend on different versions of already-loaded packages

Assume I have installed two versions of OpenMPI with matching versions of Boost by running the following:

spack install boost +mpi ^openmpi@1.10.3+slurm+verbs+pmi
spack install boost +mpi ^openmpi@1.8.8+slurm+verbs+pmi

Now if a user does the obvious thing

spack load openmpi
spack load boost

they end up with all matching modules loaded:

Currently Loaded Modulefiles:
  1) spack                              4) boost-1.62.0-gcc-4.9.4-bb3jpvq
  2) openmpi-1.8.8-gcc-4.9.4-gyda6cg    5) boost-1.62.0-gcc-4.9.4-5kz43yc
  3) openmpi-1.10.3-gcc-4.9.4-mnqdpir

This is of course never the intended action. To get the correct set of modules, a user will need to run commands like these instead:

spack load openmpi@1.10.3
spack load boost ^openmpi@1.10.3

spack find --deps can tell me what dependencies a package is linked to, but constructing dependency specs is not something you’d want to bother a novel user with if you can avoid it (also, they shouldn’t need to care whether they’re using OpenMPI 1.10.1 or 1.10.3). I think this could be made easier on the spack load level:

  1. Since loading the same package in multiple versions or with multiple different dependencies never makes sense, first of all, all spack-generated module files should declare conflicts against all modules with the same name.
  2. It should be possible to specify a default version that will be loaded if no version number is specified (e.g. spack load openmpi could translate into spack load openmpi@1.10.3 or spack load openmpi@1.8.8 depending on configuration). Similar to how spack install openmpi will always install the most recent version, except it should be configurable to a different version. Maybe etc/spack/packages.yaml can already be used for this purpose.
  3. If a dependency module (e.g. openmpi@1.10.3) is already loaded, then loading the dependent without further dependency specification (e.g. spack load boost) should automatically load the version of the dependent that is linked against the loaded dependency (in this example, boost ^openmpi@1.10.3)

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 24 (24 by maintainers)

Commits related to this issue

Most upvoted comments

@alalazo, sorry for the late replies…

Do you have an example of such complex conflicts that comes e.g. from a use-case?

This is mainly a concern when you need multiple libraries (e.g. Scalapack and FFTW) that depend on a common library (e.g. MPI) or when you both need a library (e.g. Boost) and one of its dependencies (e.g. Python). Assuming I have the following packages installed:

  • boost @1.62 +python ^python@2.7
  • boost @1.62 +python ^python@3.4
  • boost @1.62 +python ^python@3.5
  • boost @1.65 +python ^python@2.7
  • boost @1.65 +python ^python@3.5
  • boost @1.65 +python ^python@3.6

Then I’d like declare that each boost module conflicts with all python modules except for the one that it was built against. If I don’t have these conflicts declared, then users who accidentally load boost @1.62 +python ^python@2.7 and python@3.6 and try to use Boost.Python in their own code get linker errors. If this was caught by the module system, the error would be clearer to the user.

Similarly, assuming I have the following packages installed:

  • netlib-scalapack ^openmpi@3
  • netlib-scalapack ^openmpi@2
  • fftw ^openmpi@3
  • fftw ^openmpi@2

Then I’d like to see an error message if fftw ^openmpi@3 and netlib-scalapack ^openmpi@2 are loaded together.

I guess this mostly affects Python and MPI as these don’t allow you to use multiple versions of the same library from within the same process due to their complex internal state.

I think this is solved by using lmod (a.k.a. hierarchical) module files.

You may be right. Adding python and mpi to the hierarchy should suffice. I’m currently trying that.