scipy: BUG: meson fails to properly detect numpy, pybind11 and (sort of) pythran when cross compiling

The new dependency detection in scipy/meson.build works suitably for native builds of SciPy, but does not correctly detect build-arch dependencies for cross compilation. The meson rules invoke the host Python interpreter, there called py3, to import the numpy, pybind11 and pythran packages (all installed for the host arch) and deduce the proper paths for includes and, in the case of NumPy, infer a search path for the npymath and npyrandom libraries.

NOTE: the Pythran search is incorrect but doesn’t really cause a problem because the it is only used on the host to render files; the include path returned by the problematic search is never referenced by any subsequent meson build logic. The entire incdir_pythran block could be removed without adverse effects.

If the requisite dependencies are installed for the host arch (whether or not they are also installed for the build arch), the detection will return paths for the host arch, which might cause subtle problems in header files and fails outright when the linker attempts to link object files for the build arch with the native npymath or npyrandom libraries. If the dependencies are installed only for the build arch, the interpreter will fail entirely and meson will never even configure the build.

Resolution attempts

I have tried a couple simple workarounds, to no avail:

1. Install only the dependencies only for the build arch. On Void Linux, we set a lot of environment variables to tell the host Python to use the sysconfig data for the build arch and also add the build root to PYTHONPATH, allowing the host Python to find these modules and grabbing relevant information (field sizes, shlib suffixes, etc.) for the build arch rather than the host. This would work with the existing meson detection for pybind11 and pythran, but does not work for numpy because import numpy triggers a bunch of shared object loads and the build arch libraries are incompatible with the host. (A more targeted import, such as from numpy.__config__ import get_include, might work in this situation, but I haven’t bothered to try.)
2. Install dependencies for host and build arches, use the detection to find paths to the host, but then manually prepend the build root to the returned paths. This almost works on Void Linux because we install packages for the build arch under a /usr/<triple> prefix that otherwise mirrors the native layout. In generally, most of the compilation commands include the right -I flags to find headers for the build arch. However, some commands still include paths to the host interpreter, and the find_library calls to identify npymath and npyrandom somehow still pick up the host versions and trigger a linker failure. I don’t know enough about how meson sets the Python environment when searching for it to understand how these host paths are creeping in or why find_library still seems to prefer the host paths even though I add the correct paths to the search paths in that function. (I wouldn’t really expect find_library to dig into the numpy tree to find the libraries, so it seems an additional search path is creeping in before I add one explicitly.)

Possible fixes

Although I’m speculating, it seems a few approaches could be taken to resolve this issue, in order of decreasing “niceness”:

1. Convince numpy (and, probably, pybind11) to ship pkg-config files. This is probably desirable, was mentioned in the [related meson issue], and sidesteps a lot of problems. The trouble with pybind11 is that it wants to be entirely self-contained within the Python package tree; however, even if it ships a .pc file within its package tree instead of in a system-specific path, Void can probably work around the issue with relative ease. (Void already wraps pkg-config for cross builds so that it loads descriptors for the build arch and manipulates the paths appropriately.) The determination for pythran should just be dropped altogether (also, rather than invoking a Python interpreter to read SCIPY_USE_PYTHRAN from the environment, Pythran should just be a meson build option). The trouble here is backwards compatibility; if SciPy will build with old versions of NumPy or pybind11, it will still need fallback detection. Hence…
2. Existing logic can be improved, even if only as a fallback for older versions of dependencies. I’m not sure what this should look like, but reading variables from the environment might make sense (e.g., NUMPY_ROOT, PYBIND11_INCLUDE_DIR); when these variables are defined, they are used as-is; otherwise, the existing interpreter invocations can provide sensible defaults for native builds. Of course, this assumes that find_library can be made to find NumPy libraries for the build arch even though it now prefers the host versions.
3. The existing search for a Python interpreter could allow a custom path rather than always using the default, which seems to be the same interpreter that is running meson. This might allow some clever wrapping of the interpreter but is probably an incomplete (and maybe completely ineffective) solution. For example, no amount of sensible wrapping will allow print(numpy.get_include()) to dump some modified path; however, a successful from numpy.__config__ import get_include using the build-arch NumPy might make this workable if ugly.

Related issue

This issue was opened in response to https://github.com/mesonbuild/meson/issues/9598#issuecomment-1201475862 as a means to track SciPy specifics and provide a link target for inclusion in https://github.com/scipy/scipy/issues/14812.

cc: @rgommers