scipy: DOC: linking to custom BLAS/LAPACK locations is not clear

I’ll have a go at cleaning up the docs today. There’s also still a bunch of places with very detailed explanations on distutils usage; I’ll replace those with “look at examples in current meson.build files”.

We had a good discussion at the community meeting with both some new contributors and some old hands. The most popular idea was to replace the “Getting Started” tile on the front page of the docs (http://scipy.github.io/devdocs/), which is basically empty, with a dedicated “Building from source” section. We’d then still have four tiles:

User guide                API reference

Building from source      Developer guide

I probably won’t get started on that this week, but it’d be good to hear any concerns with that plan.

You mean that we could have a page which is just the output of all help commands?

No, quite the opposite. This always goes out of date, as the current state of our docs shows. I want to explain the principle only, and then say something like “for any details on build options and how to use them, see dev.py build --help”.

or even discuss it during our next community meeting.

next community meeting sounds good to me

Related question to maintainers: should we clean the docs now? The “advanced Meson” page is quite difficult to find, and the setup.py content basically is no longer relevant even though it’s still possible to trigger that build (but not via pip for example).

I totally see and feel the pain here, but if it helps: I’m quite confident we’re near the end of that very long tunnel. Most projects and all build tools (except Poetry, which will follow soon) have adopted PEP 517. For compiled code, everyone is dropping distutils/setuptools at a good pace, and we will have only two serious general-purpose build systems left: Meson/meson-python and CMake/scikit-build-core. Those are not that dissimilar and are even collaborating on design of UX and features.

Ok, I’ll try and adapt to scikit-build-core then. More work 😃

Could I suggest that the Note in https://docs.scipy.org/doc/scipy/dev/contributor/meson_advanced.html#select-a-different-blas-or-lapack-library is complimented by the filename of the pkg-config file and the resulting meson commandline for building it. It is relatively clear, but… 😃

Thanks for the suggestion! I’ll fix that. Also, I haven’t forgotten that you’re the author of part of the more advanced BLAS/LAPACK selection knobs in numpy.distutils (thanks again!). If I may ping you for a review of the docs/behavior of the like-for-like replacements in Meson once I’m done with those, it’d be great to get your input on that.

Happy to help. 😃
And yes, please do ping me!

Another thing, on the documentation pages it is not easy to go back and forth in the pages

Agreed, I want to fix this in one go with my proposed restructure above. The sidebar content will also be much more organized then.

Great!

It would also be nice to discuss how to change CFLAGS etc.

Thanks, I’ll add that. It should be included in Build types (debug/release builds, optimization levels, etc.)

For the basics:

* Meson understands all the most standard env vars, like `CC`, `FC`, `CFLAGS`, `LDFLAGS`

* It uses CLI flags for everything else, and those can be passed through `config_settings` when using `pip`/`build`

* When using `dev.py`, that level of customization is available only for the most commonly used things, e.g. `python dev.py build --werror --debug` does what you expect there (selects a debug build and passes `-Werror` in a compiler-independent way)
  
  * If you want to get fancy beyond what `dev.py build --help` shows, you can separately run `meson setup build <options-you-want>` followed by `dev.py build`. I'll admit that this is okay but not ideal, and it could benefit from a way to directly pass any flag through to `meson setup` (e.g., add `dev.py build --setup-arg <your-desired-arg-here>`).

Ok.

meson -Dc_args='-g' ... CFLAGS="-O3" python dev.py

I hope the above answer helps. I’ll note that you should not pass -g or -O3 like that with Meson, it has better CLI flags for this. You want to do it in one line ideally:

* what works right now is: `CFLAGS="-O3" python dev.py build --debug`

* what it should become is one of:
  
  * `python dev.py build --debug --optimization 3` (for an option like `-O3` that is common enough to have its own flag)
  * `python dev.py build --debug --setup-arg b_lto=true` (e.g., for enabling link-time-optimization, or any other [Meson build option](https://mesonbuild.com/Builtin-options.html#base-options))

Yes, this would be great to have! 😃
It was just a contrived example 😃

Don’t get me wrong I struggled to build scipy on windows for months and I’m supposed to be a maintainer. So I hope you don’t think I’m being dismissive. I really mean it when I say these are brand new tools we are dealing with and meson team was exceptionally receptive. This doesn’t mean that all is unicorns and rainbows.

Thanks for this sentiment. I am continuously struggling to find a path in the package nightmare of python. There are too many tools and too few standards… 😦 I know this isn’t easy and I can see this frustration across the Python community. I truly hope that the coming years will try to manage a more standardized approach, pep 517 + friends. And then meson + scikit-build and others are another layer of complexity. I am not blaming anybody, but I can just see that the obfuscation and very diverse avocations of different build-tools, ends up being a problem for the small dev’s who cannot figure out what is the endorsed approach.

However we have to start with a minimum number of assumptions to make sure certain things are, at least, in place. Then we would consoder the edge cases slowly.

Agreed!

Like I mentioned there are certain options in meson to achieve custom linking and dependencies but our tools are not ready yet. So it is instrumental that you fiddle with it too. Or someone who knows how to do it tells how it is. But we are not ready to document what’s allowed and what’s not.

Ok. I will do then. 😃

Let me note one thing about meson that is not clear for builders. Environment variables. One expects that env-vars are obeyed, CC, CFLAGS. Now meson is very particular here to never allowing env-vars. I understand their concerns and decisions, but perhaps this needs a clarification in the documentation (here as well). References: https://github.com/mesonbuild/meson/issues/9 and https://github.com/mesonbuild/meson/pull/2796 Yet, some of the flags are still read: https://github.com/mesonbuild/meson/issues/4664 Finding this in the documentation is not easy… 😦

I have tried to understand this page: https://docs.scipy.org/doc/scipy/dev/contributor/building_faq.html#how-do-i-use-a-custom-blas-distribution-on-linux

Could you please have a look at https://docs.scipy.org/doc/scipy/dev/contributor/meson_advanced.html#select-a-different-blas-or-lapack-library instead? That should be the right page, and I think it should answer your questions. If not, please point out the gaps and I’ll try to address them.

Yes, this is much better, thanks! Could I suggest that the Note in https://docs.scipy.org/doc/scipy/dev/contributor/meson_advanced.html#select-a-different-blas-or-lapack-library is complimented by the filename of the pkg-config file and the resulting meson commandline for building it. It is relatively clear, but… 😃

Another thing, on the documentation pages it is not easy to go back and forth in the pages. For instance, if I just follow your link to the advanced meson page, I cannot easily find this https://docs.scipy.org/doc/scipy/dev/contributor/meson.html, no links to that page? And vice-versa. No link from the simple -> advanced. There is only an “on this page” in the right block.

It seems to me that the section structure is missing, somehow? (this is a generic problem I think)

Unfortunately we have two sets of build docs right now - we’ll get rid of the setup.py related ones very soon; hopefully it’ll be easier to find then.

That would be great! The older versions documentations still works, so one could link to prior versions documentations in a simple statement where it is then much clearer that it is deprecated. Basically what @tupul said 😃

Fixing this up completely is a little messy. I’m going to simplify the “build from source” docs a lot. We’ve got the same instructions (or different variants) in about ten different places. The “system-level dependencies” part is quite nice, and then it goes downhill quickly. I’m going to get rid of the “this is Meson, and here is how it compares to the old thing” parts too. We can simplify it down to:

1. System dependencies and how to install them (already in good shape)

2. Install from source for production with `pip` or local development with `dev.py` (already in good shape too)

3. Relevant build customizations users may need:


* BLAS/LAPACK selection

* Compiler selection

* Build types (debug/release builds, optimization levels, etc.)

* Controlling the Fortran ABI used

* Cross compilation


4. Info for packagers/redistributors

5. Background content on how Meson works and FAQ entries (trim that down, but some stuff is relevant)

There is very little of use left in the macOS/Windows/Ubuntu/Conda parts, once the “system dependencies” part is done, everything else is effectively the same.

It would also be nice to discuss how to change CFLAGS etc. It is not clear to me whether the CFLAGS from meson build-setup or from the dev.py file are used. So if I do:

meson -Dc_args='-g' ...
CFLAGS="-O3" python dev.py ...

what would happen where?

Thanks for all of the discussion!