meson-python: Build failures using meson-python 0.11.0 with cibuildwheel

On Windows, building for win32 fails with

Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\cibw-run-dbnhe2vj\cp37-win32\build\venv\Scripts\python.exe)
Need python for 64-bit, but found 32-bit
        
..\..\meson.build:24:3: ERROR: Python dependency not found

it may be a cibuildwheel issue or a Meson issue. Full log here https://github.com/dnicolodi/python-siphash24/actions/runs/3517527064/jobs/5895402010

On macOS, building succeeds but test fail. The test pass when the package is not built with cibuildwheel. No idea what is going on as it seems that the test process segfaults. Maybe a linking issue. Full log here https://github.com/dnicolodi/python-siphash24/actions/runs/3517527064/jobs/5895402097

On Linux, linking issue:

FAILED: siphash24.cpython-37m-x86_64-linux-gnu.so
        cc  -o siphash24.cpython-37m-x86_64-linux-gnu.so siphash24.cpython-37m-x86_64-linux-gnu.so.p/meson-generated_siphash24.pyx.c.o -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group subprojects/libcsiphash-1/src/libcsiphash-1.a -L/opt/_internal/cpython-3.7.13/lib -lpython3.7m -Wl,--end-group
        /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -lpython3.7m

it may be Python 3.7 specific. Again not seen when building without cibuildwheel, maybe related to the setup used in the manylinux container. Full log here https://github.com/dnicolodi/python-siphash24/actions/runs/3517527064/jobs/5895401917

EDIT: a summary of the actual issue is in https://github.com/mesonbuild/meson-python/issues/222#issuecomment-1323610463, and a Meson issue is linked from https://github.com/mesonbuild/meson-python/issues/222#issuecomment-1324214338. There is also useful discussion on cross-compiling in the comments below.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 58 (58 by maintainers)

Commits related to this issue

Most upvoted comments

Side comment: When Meson fails, we should raise SystemExit (like setuptools usually does), not throw an exception. Pip/build report a mess when there’s an uncaught exception from the build hooks, while SystemExit is handled gracefully without nearly as many extra reports about missing build hooks and changed exceptions. The actual failure is printed by Meson.

If both MSVC and the MingGW compilers are on the path, which one should Meson choose, and why? Which one should other build systems choose, and why?

That rather scares me - is Meson being tested on Python 3.12a2? Distutils has been removed there. I thought the point of moving to Meson was to be ready for distutils’ removal?

Meson historically relies on distutils for 2 things:

  • figuring out whether the found python thinks it needs to link to libpython
  • checking whether Debian’s patched python with patched sys.path has patched distutils to adapt to the patch, but left sysconfig broken

In both cases, the use of distutils is limited to a probe script that dumps json to stdout describing the capabilities of the found python. And in one of those cases, we were limited by “whatever Debian does”, which is beyond frustrating but there you have it.

We do intend to fix this before python 3.12 final, and ideally as soon as possible for alpha testing, but the main point is that whether or not Meson uses it isn’t something that individual packages like SciPy need to care about.

But “be ready for distutils’ removal” is a complex topic, and the main issue that many meson-python adapters have is less about the removal of distutils from the stdlib, and more about what that means for setuptools’ release velocity, API stability, and the infeasibility of maintaining numpy.distutils. The use of either distutils or setuptools isn’t healthy for large projects especially, because of various limitations which mainly boil down to “use a real, battle-tested build system that supports introspection, parallel builds, custom build rules that aren’t just open-coded in python, etc.” Having an import distutils may or may not be a hindrance to these goals, and as it happens, isn’t one.

There’s a quick hack to make this work on python 3.12 alpha, simply install setuptools as a build dependency – it includes a copy of distutils, suitable for example for checking whether there’s a debian scheme in it 😄 and which also contains stdlib-compatible logic for determining whether libpython is expected on that OS to be linked to in extensions. This is of course a hack, not a solution, which is why it will not be needed for the final release. I’ve already been looking into what is needed here.