scipy: BUG: generate_f2py mod is called by the wrong interpreter
Describe your issue.
Following the fix in https://github.com/scipy/scipy/pull/20530 I was able to investigate my build failure with the openSUSE scipy HPC package.
It turns out the reliance on the shebang in tools/generate_f2pymod.sh following https://github.com/scipy/scipy/pull/20121 is the culprit. It calls the script with the system python3, not respecting any virtual environment or custom build interpreter: The HPC build was setup with numpy for python3.10, but called python3 from 3.11 which did not have a proper HPC numpy in its path.
Tempita may have a similar issue: https://github.com/scipy/scipy/blob/b4f3037e04bd0d312cd671b7d1d5d9ffd8923472/scipy/_build_utils/tempita.py#L1
I am able to work around it by adjusting the shebang. But please reconsider hardcoding the interpreter in the vanilla sources.
Reproducing Code Example
# HPC setup on openSUSE Tumbleweed
module load gcc/13
module load openblas
module load python310-numpy
python3.10 -m pip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./build .
## Another way o reproduce: Have numpy for a custom interpreter (here python312) but not for python3 (here python3.11) installed:
docker pull opensuse/tumbleweed
docker run -it opensuse/tumbleweed
zypper ref
zypper in python311-base python312-base python312-devel python312-numpy python312-Cython python312-devel python312-meson-python python312-pip python312-pybind11-devel python312-pythran pkg-config meson gcc-c++ gcc-fortran openblas-devel wget patch
wget https://files.pythonhosted.org/packages/source/s/scipy/scipy-1.13.0.tar.gz
tar xf scipy-1.13.0.tar.gz
cd scipy-1.13.0
wget https://github.com/scipy/scipy/pull/20530.diff
patch -p1 < 20530.diff
python3.12 -m pip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./build .
Error message
[ 93s] Processing /home/abuild/rpmbuild/BUILD/scipy-1.13.0
[ 93s] Preparing metadata (pyproject.toml): started
[ 93s] Running command Preparing metadata (pyproject.toml)
[ 94s] + meson setup /home/abuild/rpmbuild/BUILD/scipy-1.13.0 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/.mesonpy-qtpo_y7n -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/abuild/rpmbuild/BUILD/scipy-1.13.0/.mesonpy-qtpo_y7n/meson-python-native-file.ini
[ 97s] The Meson build system
...
{ 97s] Program tools/generate_f2pymod.py found: YES (/usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py)
[ 97s] Program scipy/_build_utils/tempita.py found: YES (/usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/scipy/_build_utils/tempita.py)
[ 97s] Program pythran found: YES 0.15.0 0.15.0 (/home/abuild/rpmbuild/BUILD/scipy-1.13.0/build/flavorbin/pythran)
...
[ 104s] [164/1475] Generating scipy/linalg/fblas_module with a custom command
[ 104s] FAILED: scipy/linalg/_fblasmodule.c
[ 104s] /usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/fblas.pyf.src -o scipy/linalg
[ 104s] Traceback (most recent call last):
[ 104s] File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
[ 104s] main()
[ 104s] File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
[ 104s] raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
[ 104s] RuntimeError: Processing scipy/linalg/fblas.pyf with f2py failed!
[ 104s]
[ 104s] Traceback (most recent call last):
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 23, in <module>
[ 104s] from . import multiarray
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/multiarray.py", line 10, in <module>
[ 104s] from . import overrides
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/overrides.py", line 8, in <module>
[ 104s] from numpy._core._multiarray_umath import (
[ 104s] ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'
[ 104s]
[ 104s] During handling of the above exception, another exception occurred:
[ 104s]
[ 104s] Traceback (most recent call last):
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 114, in <module>
[ 104s] from numpy.__config__ import show as show_config
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__config__.py", line 4, in <module>
[ 104s] from numpy._core._multiarray_umath import (
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 49, in <module>
[ 104s] raise ImportError(msg)
[ 104s] ImportError:
[ 104s]
[ 104s] IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
[ 104s]
[ 104s] Importing the numpy C-extensions failed. This error can happen for
[ 104s] many reasons, often due to issues with your setup or how NumPy was
[ 104s] installed.
[ 104s]
[ 104s] We have compiled some common reasons and troubleshooting tips at:
[ 104s]
[ 104s] https://numpy.org/devdocs/user/troubleshooting-importerror.html
[ 104s]
[ 104s] Please note and check the following:
[ 104s]
[ 104s] * The Python version is: Python3.11 from "/usr/bin/python3.11"
[ 104s] * The NumPy version is: "2.0.0rc1"
[ 104s]
[ 104s] and make sure that they are the versions you expect.
[ 104s] Please carefully study the documentation linked above for further help.
[ 104s]
[ 104s] Original error was: No module named 'numpy._core._multiarray_umath'
[ 104s]
[ 104s]
[ 104s] The above exception was the direct cause of the following exception:
[ 104s]
[ 104s] Traceback (most recent call last):
[ 104s] File "<frozen runpy>", line 189, in _run_module_as_main
[ 104s] File "<frozen runpy>", line 112, in _get_module_details
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 119, in <module>
[ 104s] raise ImportError(msg) from e
[ 104s] ImportError: Error importing numpy: you should not try to import numpy from
[ 104s] its source directory; please exit the numpy source tree, and relaunch
[ 104s] your python interpreter from there.
[ 104s]
[ 104s] [165/1475] Generating scipy/linalg/flapack_module with a custom command
[ 104s] FAILED: scipy/linalg/_flapackmodule.c
[ 104s] /usr/bin/python3.11 /home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/flapack.pyf.src -o scipy/linalg
[ 104s] Traceback (most recent call last):
[ 104s] File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
[ 104s] main()
[ 104s] File "/home/abuild/rpmbuild/BUILD/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
[ 104s] raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
[ 104s] RuntimeError: Processing scipy/linalg/flapack.pyf with f2py failed!
[ 104s]
[ 104s] Traceback (most recent call last):
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 23, in <module>
[ 104s] from . import multiarray
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/multiarray.py", line 10, in <module>
[ 104s] from . import overrides
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/overrides.py", line 8, in <module>
[ 104s] from numpy._core._multiarray_umath import (
[ 104s] ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'
[ 104s]
[ 104s] During handling of the above exception, another exception occurred:
[ 104s]
[ 104s] Traceback (most recent call last):
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 114, in <module>
[ 104s] from numpy.__config__ import show as show_config
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__config__.py", line 4, in <module>
[ 104s] from numpy._core._multiarray_umath import (
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/_core/__init__.py", line 49, in <module>
[ 104s] raise ImportError(msg)
[ 104s] ImportError:
[ 104s]
[ 104s] IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
[ 104s]
[ 104s] Importing the numpy C-extensions failed. This error can happen for
[ 104s] many reasons, often due to issues with your setup or how NumPy was
[ 104s] installed.
[ 104s]
[ 104s] We have compiled some common reasons and troubleshooting tips at:
[ 104s]
[ 104s] https://numpy.org/devdocs/user/troubleshooting-importerror.html
[ 104s]
[ 104s] Please note and check the following:
[ 104s]
[ 104s] * The Python version is: Python3.11 from "/usr/bin/python3.11"
[ 104s] * The NumPy version is: "2.0.0rc1"
[ 104s]
[ 104s] and make sure that they are the versions you expect.
[ 104s] Please carefully study the documentation linked above for further help.
[ 104s]
[ 104s] Original error was: No module named 'numpy._core._multiarray_umath'
[ 104s]
[ 104s]
[ 104s] The above exception was the direct cause of the following exception:
[ 104s]
[ 104s] Traceback (most recent call last):
[ 104s] File "<frozen runpy>", line 189, in _run_module_as_main
[ 104s] File "<frozen runpy>", line 112, in _get_module_details
[ 104s] File "/usr/lib/hpc/gnu13/python-numpy/2.0.0~rc1/lib64/python3.10/site-packages/numpy/__init__.py", line 119, in <module>
[ 104s] raise ImportError(msg) from e
[ 104s] ImportError: Error importing numpy: you should not try to import numpy from
[ 104s] its source directory; please exit the numpy source tree, and relaunch
[ 104s] your python interpreter from there.
[ 104s]
# The docker reproducer:
Processing /scipy-1.13.0
Running command Preparing metadata (pyproject.toml)
+ meson setup /scipy-1.13.0 /scipy-1.13.0/.mesonpy-oavlll2e -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/scipy-1.13.0/.mesonpy-oavlll2e/meson-python-native-file.ini
The Meson build system
Version: 1.4.0
Source dir: /scipy-1.13.0
Build dir: /scipy-1.13.0/.mesonpy-oavlll2e
Build type: native build
Project name: scipy
Project version: 1.13.0
C compiler for the host machine: cc (gcc 13.2.1 "cc (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]")
C linker for the host machine: cc ld.bfd 2.42.0.20240130-2
C++ compiler for the host machine: c++ (gcc 13.2.1 "c++ (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]")
C++ linker for the host machine: c++ ld.bfd 2.42.0.20240130-2
Cython compiler for the host machine: cython (cython 3.0.10)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (/usr/bin/python3.12)
Found pkg-config: YES (/usr/bin/pkg-config) 2.1.1
Run-time dependency python found: YES 3.12
Program cython found: YES (/usr/bin/cython)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Library m found: YES
Fortran compiler for the host machine: gfortran (gcc 13.2.1 "GNU Fortran (SUSE Linux) 13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]")
Fortran linker for the host machine: gfortran ld.bfd 2.42.0.20240130-2
Compiler for Fortran supports arguments -Wno-conversion: YES
Checking if "-Wl,--version-script" : links: YES
Program tools/generate_f2pymod.py found: YES (/usr/bin/python3.11 /scipy-1.13.0/tools/generate_f2pymod.py)
Program scipy/_build_utils/tempita.py found: YES (/usr/bin/python3.11 /scipy-1.13.0/scipy/_build_utils/tempita.py)
[162/1475] Generating scipy/linalg/fblas_module with a custom command
FAILED: scipy/linalg/_fblasmodule.c
/usr/bin/python3.11 /scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/fblas.pyf.src -o scipy/linalg
Traceback (most recent call last):
File "/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
main()
File "/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
RuntimeError: Processing scipy/linalg/fblas.pyf with f2py failed!
/usr/bin/python3.11: Error while finding module specification for 'numpy.f2py' (ModuleNotFoundError: No module named 'numpy')
[163/1475] Generating scipy/linalg/flapack_module with a custom command
FAILED: scipy/linalg/_flapackmodule.c
/usr/bin/python3.11 /scipy-1.13.0/tools/generate_f2pymod.py ../scipy/linalg/flapack.pyf.src -o scipy/linalg
Traceback (most recent call last):
File "/scipy-1.13.0/tools/generate_f2pymod.py", line 299, in <module>
main()
File "/scipy-1.13.0/tools/generate_f2pymod.py", line 293, in main
raise RuntimeError(f"Processing {fname_pyf} with f2py failed!\n"
RuntimeError: Processing scipy/linalg/flapack.pyf with f2py failed!
/usr/bin/python3.11: Error while finding module specification for 'numpy.f2py' (ModuleNotFoundError: No module named 'numpy')
SciPy/NumPy/Python version and system information
Scipy: 1.13.0 sdist from PyPI
Numpy: 2.0.0rc1 (Same failure with 1.26.4)
sys.version_info(major=3, minor=10, micro=14, releaselevel='final', serial=0)
scipy.show_config(): N/A (build failure)
EDIT: this bug exists only in scipy 1.13.0; it was fixed in 1.13.1 by reverting the problematic change, and the milestone of this issue is set to 1.14.0 for a different structural fix.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (15 by maintainers)
Commits related to this issue
- MAINT: vendor Tempita in `scipy/_build_utils` This is an unmodified copy from `Cython/Tempita/` as of commit `023d4af35` in Cython's master branch (21 April 2024). Tempita hasn't been maintained inde... — committed to rgommers/scipy by rgommers 3 years ago
- MAINT: vendor Tempita in `scipy/_build_utils` This is an unmodified copy from `Cython/Tempita/` as of commit `023d4af35` in Cython's master branch (21 April 2024). Tempita hasn't been maintained inde... — committed to rgommers/scipy by rgommers 3 years ago
I got it. The shebang was never
#!/usr/bin/python3. Of course#!/usr/bin/env python3resolves to the interpreter of the build venv when not running non-isolated.I don’t think that they are interfering. What happened there is that I only applied the above rules partially before, so inside
generate_f2pymod.pythere was apython3shebang and it was importingnumpy.f2py. Which happened to work whenmesonwas installed in the same env as meson-python and numpy, but not if it was in the default interpreter’s env. Or something like that.The added CI job in gh-20612 replicated the exact error you were reporting here, and it was because of the config I summarized here. Once the rules in my comment above were applied, the error went away because the
f2pyexecutable always works wherepython3 -m numpy.f2pydoesn’t work for the wrongpython3interpreter.