scipy: undefined symbol: _gfortran_stop_numeric_f08

Okay, for who reads this, do not install the conda gcc package, that is not a supported compiler: https://github.com/ContinuumIO/anaconda-issues/issues/10495#issuecomment-452485740

I finally received a full reproducer for the undefined symbol: _gfortran_stop_numeric_f08, and I think I know what the problem is. It is a SciPy (or distutils/numpy.distutils/setuptools) bug. The problem is occurring for Python extensions where we first build static libraries from Fortran code (with a Fortran compiler), and then the Python extension via C/Cython wrappers (with a C compiler). The -lgfortran link flag is going missing under some circumstances. In particular, it’s present when we have a conda environment with only the SciPy build dependencies, and it goes missing when there’s an already installed scipy package in the environment.

A successful build will look like:

/home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran -Wall -g -Wall -g -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-dev/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib build/temp.linux-x86_64-3.8/scipy/special/cython_special.o build/temp.linux-x86_64-3.8/scipy/special/sf_error.o build/temp.linux-x86_64-3.8/scipy/special/amos_wrappers.o build/temp.linux-x86_64-3.8/scipy/special/cdf_wrappers.o build/temp.linux-x86_64-3.8/scipy/special/specfun_wrappers.o build/temp.linux-x86_64-3.8/scipy/special/_cosine.o -L/home/rgommers/anaconda3/envs/scipy-dev/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib/python3.8/site-packages/numpy/core/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib/python3.8/site-packages/numpy/core/lib -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib -Lbuild/temp.linux-x86_64-3.8 -lsc_amos -lsc_cephes -lsc_mach -lsc_cdf -lsc_specfun -lopenblas -lopenblas -lnpymath -lm -lgfortran -o scipy/special/cython_special.cpython-38-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.8/link-version-scipy.special.cython_special.map

And a problematic one like:

/home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-dev/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-dev/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-dev/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/rgommers/anaconda3/envs/scipy-dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/rgommers/anaconda3/envs/scipy-dev/include build/temp.linux-x86_64-3.9/scipy/special/cython_special.o build/temp.linux-x86_64-3.9/scipy/special/sf_error.o build/temp.linux-x86_64-3.9/scipy/special/amos_wrappers.o build/temp.linux-x86_64-3.9/scipy/special/cdf_wrappers.o build/temp.linux-x86_64-3.9/scipy/special/specfun_wrappers.o build/temp.linux-x86_64-3.9/scipy/special/_cosine.o -L/home/rgommers/anaconda3/envs/scipy-dev/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib/python3.9/site-packages/numpy/core/lib -L/home/rgommers/anaconda3/envs/scipy-dev/lib/python3.9/site-packages/numpy/core/lib -Lbuild/temp.linux-x86_64-3.9 -lsc_amos -lsc_cephes -lsc_mach -lsc_cdf -lsc_specfun -lopenblas -lopenblas -lnpymath -lm -o scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.9/link-version-scipy.special.cython_special.map

Also useful to cross-check with Meson:

build scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so: fortran_LINKER scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so.p/meson-generated_scipy_special_cython_special.pyx.c.o scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so.p/_cosine.c.o scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so.p/amos_wrappers.c.o scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so.p/cdf_wrappers.c.o scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so.p/specfun_wrappers.c.o scipy/special/cython_special.cpython-39-x86_64-linux-gnu.so.p/sf_error.c.o | /home/rgommers/anaconda3/envs/scipy-dev/lib/libopenblas.so /home/rgommers/anaconda3/envs/scipy-dev/lib/python3.9/site-packages/numpy/core/include/../lib/libnpymath.a scipy/special/libamos.a scipy/special/libcdflib.a scipy/special/libcephes.a scipy/special/libmach.a scipy/special/libspecfun.a
 LINK_ARGS = -Lers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/x86_64-conda-linux-gnu/9.4.0/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/../lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../x86_64-conda-linux-gnu/9.4.0/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/lib/x86_64-conda-linux-gnu/9.4.0/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/x86_64-conda-linux-gnu/9.4.0/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../../x86_64-conda-linux-gnu/lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.4.0/../../../ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/ -L/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,-rpath-link,/home/rgommers/anaconda3/envs/scipy-dev/lib -Wl,--start-group scipy/special/libamos.a scipy/special/libcdflib.a scipy/special/libcephes.a scipy/special/libmach.a scipy/special/libspecfun.a /home/rgommers/anaconda3/envs/scipy-dev/lib/python3.9/site-packages/numpy/core/include/../lib/libnpymath.a /home/rgommers/anaconda3/envs/scipy-dev/lib/libopenblas.so -lgfortran -lm -Wl,--end-group

Here we see that the inplace build in the env where scipy was already installed is missing a link to libgfortran:

$ python setup.py develop && cd scipy/special
$ ldd cython_special.cpython-38-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffc928f0000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f99b04a2000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f99b0481000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f99b02b5000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f99b081a000)

$ ldd ~/anaconda3/envs/amd-cucim-new/lib/python3.8/site-packages/scipy/special/cython_special.cpython-38-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffd80394000)
        libgfortran.so.5 => /home/rgommers/anaconda3/envs/amd-cucim-new/lib/python3.8/site-packages/scipy/special/../../../../libgfortran.so.5 (0x00007f78448a6000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f784473b000)
        libgcc_s.so.1 => /home/rgommers/anaconda3/envs/amd-cucim-new/lib/python3.8/site-packages/scipy/special/../../../../libgcc_s.so.1 (0x00007f7844726000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f784455a000)
        libquadmath.so.0 => /home/rgommers/anaconda3/envs/amd-cucim-new/lib/python3.8/site-packages/scipy/special/../../../.././libquadmath.so.0 (0x00007f784451e000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f7844c48000)

From the above it seems clear that we need the Fortran linker and link with -lgfortran. It’s unclear what exact bug in the build system is responsible for switching from Fortran to C. I think we won’t be solving it anymore, the problem will go away when we drop the numpy.distutils build.

@rgommers spack should build scipy 1.7.3 from source specifically for usage by seaborn, which in turn is built from source for usage by the main package. All sub-dependencies should also be specifically built from source as needed in the dependency graph.

I guess what I’ll do is try a clean buildout of the “environment” on a few different supercomputers/compilers. If the problem is thoroughly reproducible as a blocker then I may have to look into it, though at least blocking a lab project means I could use project time to fix it, though that doesn’t make it fun to fix.

No, the solution is to build with the environment.yml file in the repo, and not with a larger environment which includes scipy itself. Or are you still issue hitting this on a clean env make with that file?

I am hitting this error whenever I try to build scipy over a manually built numpy (from main).

If you want to use the environment.yml in the numpy repo, can you just remove scipy from it? It should work then. If not, please send me a reproducer and I’ll see how to avoid the problem.

Also, you’re on Linux right? You should be fine using meson to build SciPy from now on.

The bleeding has stopped here I think. It has become much easier to install conda compilers, from conda-forge through the compilers package (unfortunately defaults still has the weird compiler names one cannot remember). This is what we recommend using now in the docs: http://scipy.github.io/devdocs/dev/contributor/quickerstart_conda.html

The docker container from the instructions above is totally self contained, in that it doesn’t touch any file in the host OS. When you leave the container whatever operations you’ve done in the container are forgotten. It would be possible to work on your own fork in the container:

1. Remove the last line of the Dockerfile I gave above. That line clones the main scipy repository into the docker container.
2. Rebuild the container docker build -t scipy-dev .
3. Start the docker container with

docker run -it -v /my_path_to_scipy_directory/scipy:/home/scipy scipy-dev

Here /my_path_to_scipy_directory refers to the path of your local scipy fork in the host OS. I’m presuming you have forked scipy to your account on github, then git cloned that to your own computer. When the docker container starts this directory will be mounted and made available in /home/scipy. This means whatever changes you make to files inside that directory in the container will also be made in the host OS. 4) Make your development changes (from a git feature branch). You can either do that with vim in the container, or in the host OS. 5) In the docker container you can use python runtests.py (from the /home/scipy directory) to see if the test suite still passes. 6) Once you’re happy with the changes you need to git commit those changes and git push them to your fork of scipy on github. Then you can start a pull request.

These instructions work for whatever platform Docker runs on (macOS/Linux/Windows/etc). The advantage to using Docker is that you can get a reproducible OS setup very quickly.

I’ve just built scipy on Ubuntu bionic, and ran the tests successfully. Note, my host OS is macOS, and I did the build using a brand new Docker container. I used this Dockerfile.

Instructions:

1. Install Docker, and make sure that it’s working properly.
2. Download the Dockerfile mentioned above, and rename it Dockerfile.
3. Build the docker container (from a terminal window) docker build -t scipy-dev .. This assumes that the Dockerfile is in the current directory (.). We’re calling the container scipy-dev.
4. Start up the docker container docker run -it scipy-dev. The -it flag means we want to interact with the docker container, so the command prompt should change to that of the docker container.
5. Change into the scipy git repository directory cd /home/scipy.
6. Build scipy python setup.py build
7. Install scipy pip install .
8. Change out of the scipy directory cd ...
9. Start the python interpreter, import scipy and run the tests

import scipy
scipy.test()

I’ve just followed these steps now, and managed to run the test suite successfully. You can inspect the Dockerfile to see what packages I installed from apt. No other messing about (e.g. renaming of files) was necessary. @mdhaber I’d be fine hooking up on a conference call to show you how it’s done, if you’re interested.