pip: Build dependencies doesn't use correct pinned version, installs numpy twice during build-time

Neither of those suggestions work super cleanly and are actually more difficult to understand and explain than “isolated builds are isolated”. As of today, you have 2 options: carefully pin the build dependencies, or tell pip to not do build isolation (i.e. you’ll manage the build dependencies in the environment).

Beyond that, I’m not excited by the idea of additional complexity in the dependency resolution process, that makes isolated builds depend on existing environment details – both of your suggestions require adding additional complexity to the already NP-complete problem of dependency resolution, and that code is already complex enough. And, they’re “solutions” operating with incomplete information which will certainly miss certain usecases (eg: custom compiled package that wasn’t installed via a wheel).

At the end of the day, pip isn’t going to be solving every use case perfectly, and this is one of those imperfect cases at the moment. For now, that means additional work on the user’s side, and I’m fine with that because we don’t have a good way to have the user communicate the complete complexity of build dependencies to pip.

Even with build-isolation, it’s at least downloaded twice (so it’s a bug in pip) which wouldn’t be necessary.

Both 1.19.5 and 1.20.0 are perfectly valid numpy versions to satisfy the build-dependencies, so if i instruct pip to donwload 1.19.5 - why download 1.20.0 too (and on top of that, cause a potential build-compatibility issue alongside that).

edit: I think there should be the following behaviour:

• if the build-dependency is specifically pinned (numpy==1.20.0) - then the build-installation should use that dependency, and install whatever is given otherwise in the “regular” environment.
• when it’s loosely pinned (numpy>=1.13.0) - it should use as build-dependency what’s installed in the same command - and ONLY fall back to the latest version if that dependency is not correctly installed to begin with.

Here’s a similar, but slightly different case:

• Package A
  • doesn’t have numpy in it’s pyproject.toml as it’s only a runtime dependency
  • specifies a specific version of numpy==1.19.4 in a requirements.txt
  • depends on Package B, also specified in the same requirements.txt
  • maintainer of Package A does not maintain Package B, only depends on it
• Package B
  • has a pyproject.toml listing numpy as a build dependency (has C code in the package).
  • is installed from PyPI via an sdist, so needs to be built/compiled when installed
  • can be built/compiled with a wide range of numpy versions

The above scenario produces the same results where the pinned version of numpy==1.19.4 in Package A is not used to build the dependency Package B that does need numpy. Same error results.

I had the same issue today. Since the release of numpy 1.20.0 yesterday, there is a new dimension to this problem.

For instance, I (mostly my users and CI services) usually install the package dclab with

python3 -m venv env
source env/bin/activate
pip install --upgrade pip wheel
pip install dclab[all]

dclab comes with a few cython extensions that need to be built during installation, which is a perfectly normal use-case. This is not one of those imperfect cases.

Now, the problem is that during installation of dclab, pip downloads numpy 1.20.0 and builds the extensions. But in the environment env, pip installs numpy 1.19.5 (pinned by tensorflow). When I then try to import dclab, I get this error (GH Actions):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/paul/repos/dclab/dclab/__init__.py", line 6, in <module>
    from . import definitions as dfn  # noqa: F401
  File "/home/paul/repos/dclab/dclab/definitions.py", line 4, in <module>
    from .rtdc_dataset.ancillaries import AncillaryFeature
  File "/home/paul/repos/dclab/dclab/rtdc_dataset/__init__.py", line 4, in <module>
    from .check import check_dataset  # noqa: F401
  File "/home/paul/repos/dclab/dclab/rtdc_dataset/check.py", line 10, in <module>
    from .core import RTDCBase
  File "/home/paul/repos/dclab/dclab/rtdc_dataset/core.py", line 12, in <module>
    from ..polygon_filter import PolygonFilter
  File "/home/paul/repos/dclab/dclab/polygon_filter.py", line 8, in <module>
    from .external.skimage.measure import points_in_poly
  File "/home/paul/repos/dclab/dclab/external/__init__.py", line 3, in <module>
    from . import skimage
  File "/home/paul/repos/dclab/dclab/external/skimage/__init__.py", line 2, in <module>
    from . import measure  # noqa: F401
  File "/home/paul/repos/dclab/dclab/external/skimage/measure.py", line 7, in <module>
    from .pnpoly import points_in_poly  # noqa: F401
  File "/home/paul/repos/dclab/dclab/external/skimage/pnpoly.py", line 1, in <module>
    from ._pnpoly import _grid_points_in_poly, _points_in_poly
  File "dclab/external/skimage/_pnpoly.pyx", line 1, in init dclab.external.skimage._pnpoly
    #cython: cdivision=True
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

As far as I can see, I have only three choices:

• use oldest-supported-numpy in pyproject.toml (which actually works for dclab)
• install dclab with --no-build-isolation, which I cannot really expect from my users.
• switch from tensorflow to pytorch, which is anyway used more in research as I learned today

The best solution to this problem, as far as I can see, would be for pip to be smart about choosing which version of the build dependency in pyproject.toml to install:

• if there is a numpy already installed in the environment, install the same version to build the extension
• otherwise, if the pip install command already came up with a certain version range for numpy, use the highest available version (e.g. pip install dclab[all] tensorflow would tell pip that tensorflow needs numpy 1.19.5 and so it makes sense to use that when building the extensions)

I know that pinning versions is not good, but tensorflow is doing it apparently, and many people use tensorflow.

[EDIT: found out that oldest-supported-numpy works for me]

numpy has a mechanism for dealing with this situation, that I mentioned in the first comment for this issue: oldest-supported-numpy. Anyone who isn’t using that, use that and please push your upstream libraries use that as well.

Other than that, I’ve also stated that this is a very difficult problem theoretically; even if we ignore the implementation complexities here. Looking at the votes on that comment BTW, I should note that I’d be very happy to receive a bunch of PRs that solve this without creating a special case for numpy. I think you can also submit that as your PhD thesis, while you’re at it. 😃

is there a way as a consumer of package B that I do not maintain but do depend on to control what version of a build dependency gets used by pip in the isolated build?

No, sadly this isn’t possible today, mostly because no one has stepped up to design + implement this. I’m pretty sure this has been discussed elsewhere in this tracker back when PEP 518 was being implemented, but I can’t find it now. 😦

So as a followup to my above comment, is there a way as a consumer of package B that I do not maintain but do depend on to control what version of a build dependency gets used by pip in the isolated build? Concretely, is there actually any way to control which version of numpy is used in the isolated build env for a package that lists numpy as a build dependency in its pyproject.toml?

@rgommers OK, fair enough. But I still think it’s right for build isolation to be the default, and where non-isolated builds are better, people should opt in. That’s the comment I was uncomfortable with. I agree 100% that we need better protection for users who don’t have the knowledge to set things up for non-isolated builds so that they don’t get dumped with a complex build they weren’t expecting. But I don’t want anyone to have to install setuptools before they can install some simple pure-python package that the author just hasn’t uploaded a wheel for.

There hasn’t been much discussion in this issue lately, but for future reference I want to add that this is not only an issue for numpy and its ecosystem of dependent packages, but also for other packages. In helmholtz-analytics/mpi4torch#7 we face a similar issue with pytorch and I don’t think that the purported solution of creating a meta package like oldest-supported-numpy would rectify the situation in our case, simply since pytorch is much more lenient regarding API/ABI compatibility across versions. So for me this issue mostly reads like “current build isolation implementation in pip breaks C/C++ ABI dependencies across different packages.”

To be fair, pip’s behavior probably is fully PEP517/518 compliant, since these PEPs only specify “minimal build dependencies” and how to proceed with building a single package. What we are asking for is more: We want pip to install “minimal build dependencies compatible with the to-be installed set of other packages”.

This got me thinking that given pip calls itself to install the build dependencies in build_env.py, couldn’t one add sth. like “weak constraints” (weak in the sense that build dependencies according to PEP517/518 always take precedence) that contain the selected version-specified set of the other to-be-installed packages?

However, and that is probably where the snake bites its tail, the build environments AFAIK already need to be prepared for potential candidates of to-be-installed packages? As such we would not have the final set of packages available, and even for simply iterating over candidate sets one cannot only anticipate that this can become expensive, but there are probably some nasty corner cases. @pradyunsg Is this the issue you are refering to in your comment? If so, do you have an idea on how to fix this?

Structurally this needs better tooling as either the build time numpy needs to be pinned low, or the build process needs to generate wheels with updated requirements

However none of the tools is something in pip, this is a topic for numpy, setuptools and the build backends as far as I can tell

Regarding your idea about the metadata, that is probably the big question in my opinion: Is it possible to fix this issue, maybe by implementing a good-enough heuristic in pip, that works for most cases, or does it need additional metadata to find feasible solutions.

No additional metadata is needed I believe. Right now, this example from the issue description:

pip install --no-cache numpy==1.19.5 py_find_1st==1.1.4

should error out if the runtime dependencies are correct in the py_find_1st wheel - with an understandable error message. Something like “numpy==1.19.5 and numpy>=1.24.1 (coming from py_find_1st==1.1.4) constraints are incompatible”. Bonus points for pointing to the two possible solutions: using PIP_CONSTRAINT or removing the explicit ==1.19.5 pin.

There is no way to “fix” this in pip by automatically changing something - the two constraints are actually incompatible.

To highlight the install-time issue I created a draft PR https://github.com/pypa/pip/pull/11778 that adds a (so far failing) test to the pip test collection. Maybe sb. has a good idea on how to proceed from there.

To be honest, I’ve lost the thread of what’s going on here. And a PR including just a test that claims to demonstrate “the problem”, without clearly explaining what the problem is in isolation (i.e., without expecting the reader to have followed this whole discussion) isn’t of much help here.

If someone can add a comment to the PR describing a way to reproduce the issue it’s trying to demonstrate, in terms of how to manually write a package that shows the problem, with step by step explanations, that would help a lot. I tried and failed to reverse engineer the logic of the test (the unquoted_string business lost me).

@pfmoore no typo, this really does work better without build isolation. Build isolation is a tradeoff, some things get better, some things get worse. Dealing with numpy-like ABI constraints is certainly worse (as this issue shows). There are other cases, for example when using pip install . in a conda/spack/nix env, you never want build isolation if you deal with native dependencies. Same for editable installs, that arguably should disable build isolation.

No worries, I am not planning to propose any changes to how things work today. You just have to be aware that it’s not clearcut and there are some conceptual issues with the current design.

Or maybe you’re assuming that no end user ever needs to install a package that’s available only in sdist form?

On the contrary - I do it all the time, and so do the many users whose bug reports on NumPy and SciPy I deal with.

i believe at the first level its fair to define that any package that builds binary wheels which have stricter dependencies than the source package and don’t have the wheel reflect those stricter requirements is fundamentally wrong

at the second level i think there is need for a pep that allows packages to communicate that to tools and revolvers

I don’t think there are general docs on this, as it requires specific knowledge of the individual project. To give a broad summary, though:

For runtime dependencies, pip freeze does what you want. For build dependencies, you’ll need to use --no-build-isolation, and then create a new virtual environment of your own where you do the build. Then manually extract the data from the build requirements in pyproject.toml and use that as a starting point for your build environment. Modify and test as needed until you have a working build environment (you’ll have to do this, as the whole point here is that pyproject.toml isn’t correctly capturing some details of the build environment that’s needed). Then use pip freeze to capture the build environment details, and put that in a requirements.txt file that you’ll use in future to set up the build environment. Maintain that requirements file going forward as you would with any other requirements file.

Agreed, this is a lot of manual work, but it’s basically what was needed before PEP 518 and pyproject.toml, so it shouldn’t be that surprising that you need it if pyproject.toml and isolated builds aren’t sufficient, I guess.

Interesting. There’s a leaky abstraction or two in there somewhere. A Dockerfile aims to be a repeatable build but these steps inside it:

pip install numpy==1.19.5
pip install numpy==1.19.5 cvxpy==1.1.7

now quietly build a broken Python environment just because numpy==1.20 was released.

• Why is --no-build-isolation not the default? Does the build process need to install other packages temporarily?
• Could it at least build cvxpy using the same release of numpy that’s installed and named in the current pip install command?
• If the above commands included --no-binary=numpy (which compiles numpy from source, e.g. in order to link to a specific OpenBLAS) would the cvxpy temporarily install numpy the same way? If not, could that also break the cvxpy installation?

For me it installs numpy 1.17.3; the oldest-supported-numpy package on PyPI states:

install_requires = 
	
	numpy==1.16.0; python_version=='3.5' and platform_system=='AIX'
	numpy==1.16.0; python_version=='3.6' and platform_system=='AIX'
	numpy==1.16.0; python_version=='3.7' and platform_system=='AIX'
	
	numpy==1.18.5; python_version=='3.5' and platform_machine=='aarch64'
	numpy==1.19.2; python_version=='3.6' and platform_machine=='aarch64'
	numpy==1.19.2; python_version=='3.7' and platform_machine=='aarch64'
	numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64'
	
	numpy==1.13.3; python_version=='3.5' and platform_machine!='aarch64' and platform_system!='AIX'
	numpy==1.13.3; python_version=='3.6' and platform_machine!='aarch64' and platform_system!='AIX' and platform_python_implementation != 'PyPy'
	numpy==1.14.5; python_version=='3.7' and platform_machine!='aarch64' and platform_system!='AIX' and platform_python_implementation != 'PyPy'
	numpy==1.17.3; python_version=='3.8' and platform_machine!='aarch64' and platform_python_implementation != 'PyPy'
	numpy==1.19.3; python_version=='3.9' and platform_python_implementation != 'PyPy'
	
	numpy==1.19.0; python_version=='3.6' and platform_python_implementation=='PyPy'
	numpy==1.19.0; python_version=='3.7' and platform_python_implementation=='PyPy'
	
	numpy; python_version>='3.10'
	numpy; python_version>='3.8' and platform_python_implementation=='PyPy'

I just checked with a pip install numpy==1.20.0 in my environment. Pip complains about about tensorflow being incompatible with it, but dclab imports and the tests run just fine. I assume that is because of the backwards compatibility (https://pypi.org/project/oldest-supported-numpy/):

The reason to use the oldest available Numpy version as a build-time dependency is because of ABI compatibility. Binaries compiled with old Numpy versions are binary compatible with newer Numpy versions, but not vice versa.

i don’t think you can point it to how the pyproject.toml is done.

It’s pip that’s installing numpy twice (1.20.0 for building, and 1.19.5 as final version), so this can also happen with any other package combination in theory.

It works fine if you install numpy FIRST, and then the package depending on numpy, as then pip recognizes that a compatible version is available, and doesn’t install it again.

If it wasn’t with numpy but with another random package, you couldn’t point to “oldest-supported-numpy” either.

The build-dependency is specified as "numpy>=1.13.0" - which allows every numpy > 1.13.0. Using oldest-supported-numpy might even make it worse, as according to the documentation, that would pin the build-version as numpy==1.13.0 - which would break the install completely.

In short, it’s pip that should resolve the build-dependency, detect that it’s a dependency that’s going to be installed anyway, and install numpy first (using this numpy installation for the build of the other package).

Please use numpy’s oldest-supported-numpy helper for declaring dependency on numpy in pyproject.toml.