pip: PEP-517 implementation for pip

This is an issue ticket to track discussion and planning on adding PEP-517 support. Pip 10 does support PEP-518 (with the limitation that the build tool must be a wheel - this avoids the issue of circular dependencies). The next step now is to support PEP-517. All I describe here is the result of my discussion with @dstufft at PyCon 2018 sprints.

pip for PEP-518 does (for a given package to build):

if an sdist is acquired extract it to a tree source,
get the build tool and build requirements from pyproject.toml (must contain setuptools and wheel),
for the current build environment (which is a temporary directory) install the build tool and requirements (this happens by invoking pip via a subprocess),
invoke the build command by using this temporary folder to build a wheel,
install then the wheel.

Phase 1: pip install in a PEP-517 and 518 would do:

if an sdist is acquired extract it to a tree source,
get the build tool pyproject.toml,
for the current build environment (which is a temporary directory) install the build tool (this happens by invoking pip via a subprocess),
use get_requires_for_build_sdist (or it’s wheel counterpart) to get the build requirements (this happens via invoke python within a subprocess of the build environment),
for the current build environment (which is a temporary directory) install the build requirements (this happens by invoking pip via a subprocess),
invoke the build command by using this temporary environment to build a wheel (build_wheel).
install then the wheel.

Phase 2: allow pip to build packages for distribution - pip build

this follows the same paths as above with the sole difference that allows for the user to select either wheel or sdist, and that it’s invoked from the cli (e.g. pip build . --sdist.

For phase 1 most of the things are already implemented in https://github.com/pypa/pep517 by @takluyver. It even follows the method pip uses to implement PEP-518. I suggest we just take that package as a vendored package within pip, and maybe make slight alternation to it where required. Once that’s done we can go onto phase 2.

Please show your take on this if you have any obligations, otherwise I’ll try to create a PR for this.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 52 (40 by maintainers)

Most upvoted comments

Quick update. I’ve briefly stalled on the implementation, because of various family and other commitments. I should be getting back to it in a week or two (maybe sooner depending on how things go).

pfmoore on Aug 13, 2018

To clarify further my assumption on how the two flags will behave, first note that a valid project must have at least one of setup.py and pyproject.toml.

The following cases must use PEP 517: --no-use-pep517 is invalid, and --use-pep517 is the default.
- Projects with pyproject.toml and no setup.py
- Projects where pyproject.toml has a [build-system] section containing build-backend.
The following cases can choose to use PEP 517 or not:
- Projects with only setup.py (in this case --no-use-pep517 is the default).
- Projects with both pyproject.toml and setup.py, but pyproject.toml does not contain a build-backend value (in this case --use-pep517 is the default).

When there’s a choice over using PEP 517,

With --no-use-pep517 we behave just the way we do right now (direct calls to setup.py). If we’re isolating and there are no build requirements, we assume requirements of ["setuptools", "wheel"]. If we’re not isolating, or there are explicit build requirements, it’s an error if the build environment (once created) does not include the assumed requirements.
With --use-pep517, we behave as if there were a pyproject.toml with build-backend = "setuptools.build_meta". For build requirements, we do the same as in the --no-use-pep517 case but our assumed requirements are ["setuptools>=38.2.5", "wheel"] (so that we have a version of setuptools with the backend).

Note that there’s no case where --use-pep517 is invalid. So at the point where we want to make the switch, we can start by making --use-pep517 the default everywhere, and then later we just remove the option (and the code supporting --no-use-pep518).

Also note that if a project has a pyproject.toml file, it will get full PEP 517 behaviour by default. This follows the approach we took with PEP 518, where the new behaviour was triggered by the existence of pyproject.toml, so it’s a tested route for introducing the new PEPs.

Now I just hope that I can actually implement all that! 😄

pfmoore on Jul 25, 2018

Oh yeah, thanks, I’d forgotten about that 😃 (Or more accurately, I’d forgotten it was added into this issue as part of PEP 517 - I’d always considered it as a separate thing).

By the way, it’s worth noting with regard to this comment from @pradyunsg, in particular

Should pip try to do local-dir -> sdist -> wheel unless build_sdist screeches in pain? Yes.

that the current implementation of PEP 517 does not do this. I consider the change to always build via sdist to be an independent piece of work - we could have done it ages ago long before PEP 517/518, and it always got caught up in debates about incremental builds, and tools like flit that (at the time) didn’t support sdists, etc. I still think this is worth doing, and I don’t think the arguments against it are particularly compelling, but I didn’t want to block implementing PEP 517 in order to debate that.

So if someone wants to finally switch to build-via-sdist, then I’m all for it, but it would also need a separate issue.

pfmoore on Nov 23, 2018

conda-build is using the PIP_NO_BUILD_ISOLATION environment variable with the counterintuitive, but seemingly correct value of False to disable build isolation. We’ll plan on sticking with that based on this thread, and ignore the --no-use-pep517.

msarahan on Jul 25, 2018

Looking at the basic wheel building code in pip, it seems to me that the “easy” place to start with this is pip._internal.wheel.WheelBuilder, method __build_one. I can rename __build_one as _build_one_legacy and add a _build_one_pep517 to do the same but for projects with PEP 517 information. That doesn’t seem like it would be too hard to do, and would cover the basic case.

Moving on from there, I’d need to look at the path from requirement to install, to ensure that all requirements that have pyproject.toml go through the new route - source tree or sdist -> wheel -> install. It’s at this point that we’d be eliminating all of the old “direct install” routes (or at least, marking them clearly for ultimate removal once we’ve switched fully to the sdist -> wheel route).

Once that’s done, we have the basic process sorted, and it’s just the corner cases (editable installs, …) that need tidying up.

The most interesting bit will be writing tests. I’m planning on creating a dummy backend in our test directory that can be used to exercise the new code. That way we can check that all the steps are working in isolation from any actual backend processing.

By the way, there’s a question regarding setuptools (well, a couple):

Does setuptools have a PEP 517 backend yet? Is anyone working on one, and if not, what’s the plan for that?
If a project wants to use the setuptools PEP517 backend, do they have to explicitly request it in pyproject.toml? If we don’t do anything, it seems to me that there’s no incentive for setuptools-using projects to switch from using the legacy (no [build-system].build-backend key) route, to the PEP 517 backend route - however, there’s also no provision in any of the PEPs for a default backend.

My suggestion would be:

If pyproject.toml has [build-system].build-backend, then it’s a PEP 517 style source tree and we’re done.
If pyproject.toml exists, but there’s no build-backend key, but [build-system] exists, then it’s not covered by PEP 517, so it’s up to pip how we handle that case. I suggest that we initially raise a deprecation warning saying that we’re currently invoking setup.py directly, but we will switch to PEP 517 processing assuming the setuptools backend in due course. Then, once the deprecation period has passed, we do precisely that.
If pyproject.toml does not exist (or has no [build-system] section) we issue a deprecation warning saying that we’ll be moving to processing using PEP 517 (isolation, setuptools backend). Follow up by doing that, although we may want to have a longer deprecation period for that case.

We can’t do (2) and (3) until setuptools provides a PEP 517 backend, of course.

pfmoore on Jul 24, 2018

It’s not so much about an “integrity check”, as it is about ensuring that pip install . produces the same result as building a sdist manually, then pip install <the-sdist>.

This is what I meant by “integrity check”; looking back at that comment, I guess I could have worded it better.

ISTM that both @takluyver and @pfmoore have the same position as me on this:

try local-dir -> sdist -> wheel unless sdist creation is not possible.

pradyunsg on Jun 12, 2018