scipy: BUG: generate_f2py mod is called by the wrong interpreter

Describe your issue.

Following the fix in https://github.com/scipy/scipy/pull/20530 I was able to investigate my build failure with the openSUSE scipy HPC package.

It turns out the reliance on the shebang in tools/generate_f2pymod.sh following https://github.com/scipy/scipy/pull/20121 is the culprit. It calls the script with the system python3, not respecting any virtual environment or custom build interpreter: The HPC build was setup with numpy for python3.10, but called python3 from 3.11 which did not have a proper HPC numpy in its path.

Tempita may have a similar issue: https://github.com/scipy/scipy/blob/b4f3037e04bd0d312cd671b7d1d5d9ffd8923472/scipy/_build_utils/tempita.py#L1

I am able to work around it by adjusting the shebang. But please reconsider hardcoding the interpreter in the vanilla sources.

Reproducing Code Example

# HPC setup on openSUSE Tumbleweed
module load gcc/13
module load openblas
module load python310-numpy

python3.10 -m pip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./build .

## Another way o reproduce: Have numpy for a custom interpreter (here python312) but not for python3 (here python3.11) installed:

docker pull opensuse/tumbleweed
docker run -it  opensuse/tumbleweed

zypper ref
zypper in python311-base python312-base python312-devel python312-numpy python312-Cython python312-devel python312-meson-python python312-pip python312-pybind11-devel python312-pythran pkg-config meson gcc-c++ gcc-fortran openblas-devel wget patch

wget https://files.pythonhosted.org/packages/source/s/scipy/scipy-1.13.0.tar.gz
tar xf scipy-1.13.0.tar.gz
cd scipy-1.13.0
wget https://github.com/scipy/scipy/pull/20530.diff
patch -p1 < 20530.diff
python3.12 -m pip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./build .

Error message

[   93s] Processing /home/abuild/rpmbuild/BUILD/scipy-1.13.0
[   93s]   Preparing metadata (pyproject.toml): started
[   93s]   Running command Preparing metadata (pyproject.toml)
[   94s]   + meson setup /home/abuild/rpmbuild/BUILD/scipy-1.13.0 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/.mesonpy-qtpo_y7n -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/abuild/rpmbuild/BUILD/scipy-1.13.0/.mesonpy-qtpo_y7n/meson-python-native-file.ini
[   97s]   The Meson build system


...

{   97s]   Program tools/generate_f2pymod.py found: YES (/usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py)
[   97s]   Program scipy/_build_utils/tempita.py found: YES (/usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/scipy/_build_utils/tempita.py)
[   97s]   Program pythran found: YES 0.15.0 0.15.0 (/home/abuild/rpmbuild/BUILD/scipy-1.13.0/build/flavorbin/pythran)

...

[  104s]   [164/1475] Generating scipy/linalg/fblas_module with a custom command
[  104s]   FAILED: scipy/linalg/_fblasmodule.c
[  104s]   /usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/fblas.pyf.src -o scipy/linalg
[  104s]   Traceback (most recent call last):
[  104s]     File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
[  104s]       main()
[  104s]     File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
[  104s]       raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
[  104s]   RuntimeError: Processing scipy/linalg/fblas.pyf with f2py failed!
[  104s]
[  104s]   Traceback (most recent call last):
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 23, in <module>
[  104s]       from . import multiarray
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/multiarray.py", line 10, in <module>
[  104s]       from . import overrides
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/overrides.py", line 8, in <module>
[  104s]       from numpy._core._multiarray_umath import (
[  104s]   ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'
[  104s]
[  104s]   During handling of the above exception, another exception occurred:
[  104s]
[  104s]   Traceback (most recent call last):
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 114, in <module>
[  104s]       from numpy.__config__ import show as show_config
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__config__.py", line 4, in <module>
[  104s]       from numpy._core._multiarray_umath import (
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 49, in <module>
[  104s]       raise ImportError(msg)
[  104s]   ImportError:
[  104s]
[  104s]   IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
[  104s]
[  104s]   Importing the numpy C-extensions failed. This error can happen for
[  104s]   many reasons, often due to issues with your setup or how NumPy was
[  104s]   installed.
[  104s]
[  104s]   We have compiled some common reasons and troubleshooting tips at:
[  104s]
[  104s]       https://numpy.org/devdocs/user/troubleshooting-importerror.html
[  104s]
[  104s]   Please note and check the following:
[  104s]
[  104s]     * The Python version is: Python3.11 from "/usr/bin/python3.11"
[  104s]     * The NumPy version is: "2.0.0rc1"
[  104s]
[  104s]   and make sure that they are the versions you expect.
[  104s]   Please carefully study the documentation linked above for further help.
[  104s]
[  104s]   Original error was: No module named 'numpy._core._multiarray_umath'
[  104s]
[  104s]
[  104s]   The above exception was the direct cause of the following exception:
[  104s]
[  104s]   Traceback (most recent call last):
[  104s]     File "<frozen runpy>", line 189, in _run_module_as_main
[  104s]     File "<frozen runpy>", line 112, in _get_module_details
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 119, in <module>
[  104s]       raise ImportError(msg) from e
[  104s]   ImportError: Error importing numpy: you should not try to import numpy from
[  104s]           its source directory; please exit the numpy source tree, and relaunch
[  104s]           your python interpreter from there.
[  104s]
[  104s]   [165/1475] Generating scipy/linalg/flapack_module with a custom command
[  104s]   FAILED: scipy/linalg/_flapackmodule.c
[  104s]   /usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/flapack.pyf.src -o scipy/linalg
[  104s]   Traceback (most recent call last):
[  104s]     File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
[  104s]       main()
[  104s]     File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
[  104s]       raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
[  104s]   RuntimeError: Processing scipy/linalg/flapack.pyf with f2py failed!
[  104s]
[  104s]   Traceback (most recent call last):
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 23, in <module>
[  104s]       from . import multiarray
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/multiarray.py", line 10, in <module>
[  104s]       from . import overrides
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/overrides.py", line 8, in <module>
[  104s]       from numpy._core._multiarray_umath import (
[  104s]   ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'
[  104s]
[  104s]   During handling of the above exception, another exception occurred:
[  104s]
[  104s]   Traceback (most recent call last):
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 114, in <module>
[  104s]       from numpy.__config__ import show as show_config
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__config__.py", line 4, in <module>
[  104s]       from numpy._core._multiarray_umath import (
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 49, in <module>
[  104s]       raise ImportError(msg)
[  104s]   ImportError:
[  104s]
[  104s]   IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
[  104s]
[  104s]   Importing the numpy C-extensions failed. This error can happen for
[  104s]   many reasons, often due to issues with your setup or how NumPy was
[  104s]   installed.
[  104s]
[  104s]   We have compiled some common reasons and troubleshooting tips at:
[  104s]
[  104s]       https://numpy.org/devdocs/user/troubleshooting-importerror.html
[  104s]
[  104s]   Please note and check the following:
[  104s]
[  104s]     * The Python version is: Python3.11 from "/usr/bin/python3.11"
[  104s]     * The NumPy version is: "2.0.0rc1"
[  104s]
[  104s]   and make sure that they are the versions you expect.
[  104s]   Please carefully study the documentation linked above for further help.
[  104s]
[  104s]   Original error was: No module named 'numpy._core._multiarray_umath'
[  104s]
[  104s]
[  104s]   The above exception was the direct cause of the following exception:
[  104s]
[  104s]   Traceback (most recent call last):
[  104s]     File "<frozen runpy>", line 189, in _run_module_as_main
[  104s]     File "<frozen runpy>", line 112, in _get_module_details
[  104s]     File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 119, in <module>
[  104s]       raise ImportError(msg) from e
[  104s]   ImportError: Error importing numpy: you should not try to import numpy from
[  104s]           its source directory; please exit the numpy source tree, and relaunch
[  104s]           your python interpreter from there.
[  104s]


# The docker reproducer:


Processing /scipy-1.13.0
  Running command Preparing metadata (pyproject.toml)
  + meson setup /scipy-1.13.0 /scipy-1.13.0/.mesonpy-oavlll2e -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/scipy-1.13.0/.mesonpy-oavlll2e/meson-python-native-file.ini
  The Meson build system
  Version: 1.4.0
  Source dir: /scipy-1.13.0
  Build dir: /scipy-1.13.0/.mesonpy-oavlll2e
  Build type: native build
  Project name: scipy
  Project version: 1.13.0
  C compiler for the host machine: cc (gcc 13.2.1 "cc (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]")
  C linker for the host machine: cc ld.bfd 2.42.0.20240130-2
  C++ compiler for the host machine: c++ (gcc 13.2.1 "c++ (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]")
  C++ linker for the host machine: c++ ld.bfd 2.42.0.20240130-2
  Cython compiler for the host machine: cython (cython 3.0.10)
  Host machine cpu family: x86_64
  Host machine cpu: x86_64
  Program python found: YES (/usr/bin/python3.12)
  Found pkg-config: YES (/usr/bin/pkg-config) 2.1.1
  Run-time dependency python found: YES 3.12
  Program cython found: YES (/usr/bin/cython)
  Compiler for C supports arguments -Wno-unused-but-set-variable: YES
  Compiler for C supports arguments -Wno-unused-function: YES
  Compiler for C supports arguments -Wno-conversion: YES
  Compiler for C supports arguments -Wno-misleading-indentation: YES
  Library m found: YES
  Fortran compiler for the host machine: gfortran (gcc 13.2.1 "GNU Fortran (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]")
  Fortran linker for the host machine: gfortran ld.bfd 2.42.0.20240130-2
  Compiler for Fortran supports arguments -Wno-conversion: YES
  Checking if "-Wl,--version-script" : links: YES
  Program tools/generate_f2pymod.py found: YES (/usr/bin/python3.11 /scipy-1.13.0/tools/generate_f2pymod.py)
  Program scipy/_build_utils/tempita.py found: YES (/usr/bin/python3.11 /scipy-1.13.0/scipy/_build_utils/tempita.py)

[162/1475] Generating scipy/linalg/fblas_module with a custom command
  FAILED: scipy/linalg/_fblasmodule.c
  /usr/bin/python3.11 /scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/fblas.pyf.src -o scipy/linalg
  Traceback (most recent call last):
    File "/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
      main()
    File "/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
      raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
  RuntimeError: Processing scipy/linalg/fblas.pyf with f2py failed!

  /usr/bin/python3.11: Error while finding module specification for 'numpy.f2py' (ModuleNotFoundError: No module named 'numpy')

  [163/1475] Generating scipy/linalg/flapack_module with a custom command
  FAILED: scipy/linalg/_flapackmodule.c
  /usr/bin/python3.11 /scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/flapack.pyf.src -o scipy/linalg
  Traceback (most recent call last):
    File "/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
      main()
    File "/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
      raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
  RuntimeError: Processing scipy/linalg/flapack.pyf with f2py failed!

  /usr/bin/python3.11: Error while finding module specification for 'numpy.f2py' (ModuleNotFoundError: No module named 'numpy')

SciPy/NumPy/Python version and system information

Scipy: 1.13.0 sdist from PyPI
Numpy: 2.0.0rc1 (Same failure with 1.26.4)

sys.version_info(major=3, minor=10, micro=14, releaselevel='final', serial=0)

scipy.show_config(): N/A (build failure)

EDIT: this bug exists only in scipy 1.13.0; it was fixed in 1.13.1 by reverting the problematic change, and the milestone of this issue is set to 1.14.0 for a different structural fix.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (15 by maintainers)

Commits related to this issue

Most upvoted comments

I got it. The shebang was never #!/usr/bin/python3. Of course #!/usr/bin/env python3 resolves to the interpreter of the build venv when not running non-isolated.

I don’t think that they are interfering. What happened there is that I only applied the above rules partially before, so inside generate_f2pymod.py there was a python3 shebang and it was importing numpy.f2py. Which happened to work when meson was installed in the same env as meson-python and numpy, but not if it was in the default interpreter’s env. Or something like that.

The added CI job in gh-20612 replicated the exact error you were reporting here, and it was because of the config I summarized here. Once the rules in my comment above were applied, the error went away because the f2py executable always works where python3 -m numpy.f2py doesn’t work for the wrong python3 interpreter.