spack: intel-mkl recipe is broken for LANL HPC machines
Problem Statement
It looks like commit db657d938d38775e4364a6917cc78cb9cdb0b133 on Aug 16 changed the recipe for intel-mkl in a way that prevents the use intel-mkl on LANL HPC machines. The current recipe wants to set MKLROOT to self.prefix.compilers_and_libraries.linux.mkl.lib.intel64, but that is an invalid path for LANL machines. The correct path for us is self.prefix.lib.intel64.
Background/Context
On LANL HPC machines, intel-mkl is provided as an external package via the following entry in packages.yaml:
packages:
intel-mkl:
version: [11.4.1]
modules:
intel-mkl@11.4.1: mkl/11.4.1
The modulefile mkl/11.4.1 provides the following information:
conflict("mkl")
whatis("Intel Math Kernel Library 11.4.1 ")
whatis("Default settings for arch=intel64, 4-byte integers ")
setenv("MKLROOT","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017.1.132/linux/mkl")
prepend_path("LD_LIBRARY_PATH","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017/linux/mkl/lib/intel64")
prepend_path("LIBRARY_PATH","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017/linux/mkl/lib/intel64")
prepend_path("INCLUDE","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017.1.132/linux/mkl/include")
prepend_path("FPATH","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017.1.132/linux/mkl/include")
prepend_path("CPATH","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017.1.132/linux/mkl/include")
prepend_path("MANPATH","/usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017.1.132/linux/man/common")
It looks like get_path_from_module (found at module_cmd.py) extracts the intel-mkl prefix from LD_LIBRARY_PATH provided by the module file (i.e.: /usr/projects/hpcsoft/toss3/common/intel-clusterstudio/2017.1.024/compilers_and_libraries_2017/linux/mkl/) and intel-mkl/package.py appends compilers_and_libraries.linux.mkl.lib.intel64 to that path. This is clearly the wrong behavior for LANL systems. The following patch provides a work around for LANL HPC machines, but breaks the existing design.
diff --git a/var/spack/repos/builtin/packages/intel-mkl/package.py b/var/spack/repos/builtin/packages/intel-mkl/package.py
index 9a3b9b9..185444a 100644
--- a/var/spack/repos/builtin/packages/intel-mkl/package.py
+++ b/var/spack/repos/builtin/packages/intel-mkl/package.py
@@ -98,7 +98,7 @@ class IntelMkl(IntelPackage):
omp_threading = ['libiomp5']
if sys.platform != 'darwin':
- omp_root = prefix.compilers_and_libraries.linux.lib.intel64
+ omp_root = prefix.lib.intel64
else:
omp_root = prefix.lib
omp_libs = find_libraries(
@@ -114,7 +114,7 @@ class IntelMkl(IntelPackage):
# TODO: TBB threading: ['libmkl_tbb_thread', 'libtbb', 'libstdc++']
if sys.platform != 'darwin':
- mkl_root = prefix.compilers_and_libraries.linux.mkl.lib.intel64
+ mkl_root = prefix.lib.intel64
else:
mkl_root = prefix.mkl.lib
@@ -166,7 +166,7 @@ class IntelMkl(IntelPackage):
integer = 'ilp64' if '+ilp64' in self.spec else 'lp64'
mkl_root = self.prefix.mkl.lib if sys.platform == 'darwin' else \
- self.prefix.compilers_and_libraries.linux.mkl.lib.intel64
+ self.prefix.lib.intel64
shared = True if '+shared' in self.spec else False
@@ -184,8 +184,8 @@ class IntelMkl(IntelPackage):
mkl_lib = self.prefix.mkl.lib
mkl_root = self.prefix.mkl
else:
- mkl_lib = self.prefix.compilers_and_libraries.linux.mkl.lib.intel64
- mkl_root = self.prefix.compilers_and_libraries.linux.mkl
+ mkl_lib = self.prefix.lib.intel64
+ mkl_root = self.prefix
spack_env.set('MKLROOT', mkl_root)
spack_env.append_path('SPACK_COMPILER_EXTRA_RPATHS', mkl_lib)
Next Steps
I’m not sure how to resolve the different ways in which intel-mkl is installed. Thoughts?
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 18 (18 by maintainers)
I hate Intel. There installation directories change with every release, making it next to impossible to make a single package that works for everyone. I chose directories that worked for me when I installed things with Spack. If you can find a more generic way of locating the correct directories that will work for everyone, that would obviously be better. But before I changed things, many of the directories (including MKL and MPI) were incorrect for my Spack-installed versions.
sorry to chime in late, but as far as I can tell, the standard for all Intel installers (parallel studio, cluster studio, and standalone mkl) is to install in the prefix.compilers_and_libraries.linux.mkl.lib.intel64. I can see this from the 2016, 2017, and 2018 versions. Whatever is being done at LANL and on Cray systems is customized. I can tell you that for our installation here at LLNL (which I control and do outside of Spack), I add symlinks to prefix.lib, but ultimately they point to the compiler_and_libraries… directory.
That said, I think what the Spack package is doing is technically correct and sites that have a customized external build of MKL may just have to customize their Spack builds to look in the modified location. The one compromise I might argue for is to have the Spack package create symlinks like I mentioned we do here at LLNL (into prefix.lib) and then you can set MKLROOT to that symlinked directory.
Yeah, I’m not exactly sure what we have. These are usually provided through Cray but judging from their documentation (or lack thereof) it looks like intel-parallel-studio. I’ll ask someone here if they know better.