spack: Again py-scipy with Intel issues - undefined symbols

Steps to reproduce

I am building py-scipy with Intel and I am still having problems with undefined symbols as mentioned in earlier issues that were closed as completed. The problem is that during the py-scipy install it uses ifort as Fortran compiler because of https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L124 - side note: but with the gfortran compiler flags, so that doesn’t seem right either -, but when it links the Fortran extension it uses gfortran:

...
INFO: compiling Fortran sources
  INFO: Fortran f77 compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/ifort -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
  Fortran f90 compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/ifort -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
  Fortran fix compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/ifort -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
  INFO: compile options: '-DHAVE_CBLAS -Ibuild/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate -I/gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/envs/py-scipy-intel-test/install/intel/2022.0.1/py-numpy-1.22.3-vkrg7fi/
lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include/python3.9 -c'
  INFO: ifort:f77: build/src.linux-x86_64-3.9/scipy/integrate/_test_odeint_banded-f2pywrappers.f
  INFO: ifort:f77: scipy/integrate/tests/banded5x5.f
  ifort: command line warning #10157: ignoring option '-W'; argument is of wrong type
  ifort: command line warning #10006: ignoring unknown option '-ffixed-form'
  ifort: command line warning #10006: ignoring unknown option '-fno-second-underscore'
  ifort: command line warning #10157: ignoring option '-W'; argument is of wrong type
  ifort: command line warning #10006: ignoring unknown option '-ffixed-form'
  ifort: command line warning #10006: ignoring unknown option '-fno-second-underscore'
  INFO: /gpfsm/dulocal/sles12/other/gcc/10.1.0/bin/gfortran -Wall -g -Wall -g -shared build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate/_test_odeint_bandedmodule.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate/fortranobject.o build/temp.linux-x86_64-3.9/scipy/integrate/tests/banded5x5.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate/_test_odeint_banded-f2pywrappers.o -L/gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/envs/py-scipy-intel-test/install/intel/2022.0.1/openblas-0.3.19-2tgjxxs/lib -Lbuild/temp.linux-x86_64-3.9 -Wl,-rpath=/gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/envs/py-scipy-intel-test/install/intel/2022.0.1/openblas-0.3.19-2tgjxxs/lib -llsoda -lmach -lopenblas -lopenblas -lgfortran -o build/lib.linux-x86_64-3.9/scipy/integrate/_test_odeint_banded.cpython-39-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.9/link-version-scipy.integrate._test_odeint_banded.map
  INFO: building 'scipy.interpolate.interpnd' extension
  INFO: compiling C sources
  INFO: C compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/icc -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include -I/discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include -fPIC -O2 -isystem /discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include -fPIC
...

Error message

python3
from scipy.stats import skew
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/stats/__init__.py", line 453, in <module>
    from ._stats_py import *
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/stats/_stats_py.py", line 38, in <module>
    from scipy.spatial.distance import cdist
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/spatial/__init__.py", line 108, in <module>
    from ._geometric_slerp import geometric_slerp
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/spatial/_geometric_slerp.py", line 9, in <module>
    from scipy.spatial.distance import euclidean
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/spatial/distance.py", line 123, in <module>
    from ..special import rel_entr
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/special/__init__.py", line 652, in <module>
    from . import _basic
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/special/_basic.py", line 15, in <module>
    from . import _specfun
ImportError: /gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/special/_specfun.cpython-39-x86_64-linux-gnu.so: undefined symbol: __svml_pow4_l9

Similar errors on other systems when using Intel.

Information on your system

Both on HPC (Nasa Discover) and on AWS EC2 with different versions of the Intel compiler (2021.4.0 and 2022.0.1), py-scipy@1.8.0.

[py-scipy-intel-test] dheinzel@discover32:/discover/nobackup/dheinzel/spack-stack/spack-stack-scipy-intel> spack debug report
* **Spack:** 0.19.0.dev0 (aea27b394b6b2dc184d8ca3718fd57308d7312e8)
* **Python:** 3.9.7
* **Platform:** linux-sles12-haswell
* **Concretizer:** clingo

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have searched the issues of this repo and believe this is not a duplicate
  • I have run the failing commands in debug mode and reported the output

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 19 (19 by maintainers)

Most upvoted comments

@rgommers first you would need to register your Intel compiler in Spack. You can run spack compilers to see a list of compilers Spack knows about. If the intel compilers are in your PATH, you can run spack compiler find to find them. You can also specify which directory to look in.

Once Spack knows about your intel compiler, installing with intel would like like spack install py-scipy %intel.

The problem is that during the py-scipy install it uses ifort as Fortran compiler because of https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L124

This is probably a bug in the Spack package build definition. Just setting F90 is not enough I think, and that gfortran flags are being used seems to confirm that. It should be something like:

python setup.py config --compiler=intel --fcompiler=ifort install

or even

# Could be `intel` or `intelem` compiler, there's quite a few flavors here in numpy.distutils
python setup.py config --compiler=intel --fcompiler=ifort build_clib --compiler=intel --fcompiler=intelem install

I’ll note that that is being taken care of for the Fujitsu compiler, but not for Intel Fortran:

https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L116-L120

@climbfuji would you be able to provide the exact spack build/install command to reproduce this?

Just saw that. I added the intelem option in the same place before I posted this issue, but that didn’t help. I am now doing essentially the same for py-numpy, too. And set LDSHARED=“icc -shared” as well. Trying to throw everything at it and see what happens.

@rgommers I was under the impression that python setup.py ... was deprecated and everything should go through a build frontend like pip instead. Happy to expand setup.cfg to set the correct options for other compilers

Ah yes, setup.cfg works for this too - and is indeed technically nicer. setup.py deprecated, yeah technically setuptools maintainers have been saying that, but it’s not going anywhere any time soon. And for this kind of argument passing I find it easier than figuring out how to get pip to pass on the options.

What is the full list of compilers that are accepted inputs, and which keys can take arbitrary values? Spack supports a lot of uncommon compilers

The way these names are defined is via the compiler_type attribute of compiler classes in numpy.distutils, e.g. https://github.com/numpy/numpy/blob/main/numpy/distutils/intelccompiler.py#L12. As you can see in that file, there’s a bunch of different names for various flavors of using the Intel compilers (I know, it’s really bad) with different ways of having Python built. Not sure what of that is relevant here - I think Spack will build Python with the same compilers, and most tweaks are for a Python built with GCC/MSVC. I think intel and ifort will do. There’s also Fortran compiler flavors at https://github.com/numpy/numpy/blob/main/numpy/distutils/fcompiler/intel.py#L32.

The arm, fujitsu and nag compilers at https://github.com/numpy/numpy/tree/main/numpy/distutils/fcompiler are probably still relevant; the rest I have no idea.

I think we probably want to make the minimal change necessary to make Intel compilers work with SciPy 1.8.x-1.9.1 here. This stuff is all going to go away when the Meson build lands. And I’m actually in the process of adding CI for ifort and MKL: https://github.com/scipy/scipy/pull/16957.

@rgommers can you take a look at this?