pip: Raise a warning when pip falls back to the "legacy" direct install code

For me this happened due to this:

virtualenvs created with virtualenv have wheel installed:

$ virtualenv venv
created virtual environment CPython3.8.2.final.0-64 in 222ms
  creator CPython3Posix(dest=/home/ran/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/ran/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ . venv/bin/activate
(venv) $ pip list
Package    Version
---------- -------
pip        20.0.2 
setuptools 46.1.3 
wheel      0.34.2 

but virtualenvs created by python -m venv don’t:

$ python3.8 -m venv venv
$ . venv/bin/activate
(venv) $ pip list
Package    Version
---------- -------
pip        19.2.3 
setuptools 41.2.0 

I don’t know the details, but I think many people create virtualenvs using python -m venv and don’t know they should install wheel. At least I didn’t know to do this until I debugged some issue a while ago. I don’t know if it makes sense, but is it feasible to have python -m venv install wheel?

The answer may be obvious, but I’ve often wondered: As far as I understand, quite a few libraries are already vendored with pip, why isn’t wheel simply one of them?

My intuition says that we’ll get people complaining, but they will be the sort of people who aren’t looking for a transition mechanism, but rather for a “make it work again” flag. So they’ll simply complain again when the --use-deprecated flag is removed.

So I’m fine with not adding a --use-deprecated option. Adding pyproject.toml is straightforward, and if users can’t do that, and can’t fix the underlying issue that makes it impossible for them to do that within the deprecation period, and have ignored everything that’s been said in various channels about looking at moving to pyproject.toml for ages now, then I’m happy saying that they should build and publish wheels manually as a workaround, or advise their users to pin pip until they work out a better fix.

If during the deprecation period we get anyone raising a serious and genuinely intractable issue, we can review and if necessary extend the deprecation period. If they don’t engage with us until it’s too late, well, I guess it’s too late 🤷

I see the RM asked that we defer it to 3.12, which seems a reasonable decision. Maybe defer creating a PR until after 3.11 final is released, then we can aim it for 3.12, but also ask if it’s acceptable for backport to 3.11.1. It probably won’t be, because CPython is extremely cautious about adding anything but bug fixes in point releases, but ensurepip is a bit different, so no harm asking (but later, when things are a bit calmer on 3.11 🙂)

What do people think about dropping setuptools and wheel from get-pip.py?

I tend to agree with @xavfernandez’s proposal in https://github.com/pypa/pip/issues/8102#issuecomment-622401675. I’d say it can be further discussed when we reach the end of the deprecation period and we have received more feedback.

I have identified 3 cases where pip falls back to setup.py install for source requirements without pyproject.toml:

1. when --no-binary is used (and --use-pep517 is not)
2. when wheel is not installed
3. when setup.py bdist_wheel fails

I propose

• to create a different deprecation message for each
• that each message would begin with Using legacy setup.py install because...
• that each deprecation would point to a different issue to get focused feedback

@pypa/pip-committers does that sound like a plan? If yes we can discuss the exact message and solution offered to users for each in separate PRs.

+1 for this.

Do we want to start the deprecation process now (and collect feedback in a deprecation issue), or simply raise awareness with louder messages?

We could raise the level from info to warning in _should_build, and/or allude to a future deprecation:

https://github.com/pypa/pip/blob/814c54f5d7e17682f9c476f5978c4d5666abb52c/src/pip/_internal/wheel_builder.py#L72-L78

pip will also fallback to legacy install when wheel building failed for other reasons, so we may want to warn there also, and encourage people to make sure that setup.py bdist_wheel works for their packages:

https://github.com/pypa/pip/blob/814c54f5d7e17682f9c476f5978c4d5666abb52c/src/pip/_internal/commands/install.py#L368-L370

Hmm… I wonder if we should extend the isolation enabling logic, from https://github.com/pypa/pip/pull/10717 to include wheel – i.e. use isolated pyproject.toml builds if the environment does not have both setuptools and wheel installed. That’s one way to avoid falling back to setup.py install in plain python -m venv environments – before we actually make the changes in Python 3.12 for https://github.com/python/cpython/issues/95299.

Hi everybody,

I’m just a regular user. I started to see this message appearing everywhere and wondered what it could mean: “Could not build wheels for *, since package ‘wheel’ is not installed”

Now, I understand that I should install the wheels package, because it will be soon mandatory, and because it could make the build faster. But that was not the case the first time I saw this message. (I thought it was just a random message because I mess with my computer or something like that).

Why not make the message more explicit like that: “<package_name> could be build faster with package ‘wheels’. We encourage you to install it”

What do you guys think?

Basically, because pip doesn’t itself use wheel, it’s the build subprocess that we call that needs wheel. Vendoring doesn’t help in that situation. (Well, it may be that we could make it work somehow, but this issue is obsoleted by pyproject.toml, so there’s a better solution for new projects).

Interesting, I’ve never heard about pyproject.toml, and I’m a full time py dev. So you might see that it doesn’t get as much attention as you want… We’re distributing source dists just because it’s the easiest way, but I understand that wheels are better.

I understand your point. You want to push the new method because of x advantages. That’s good, because we shouldn’t stuck with old stuff, but you can also see that it’s confusing, since we should try to blame the devs, not the “installers”. But yeah, you’re pip, not setuptools…

Thanks for your answers!

One minor nitpick, if we’re going to go with the “pretend there’s a pyproject.toml” route, the default backend should be setuptools.build_meta.__legacy__ instead, which mimics the non-isolated build slightly better.

Digging into this a little bit I remember that --no-binary implies falling back to setup.py install too.

So this warning would also mean deprecating that behavior of --no-binary. Which is fine with me, as installing from source should not preclude building a wheel locally.

Agreed.

I think users who may be most impacted are application authors who are (perhaps unknowingly) on the setup.py install code path and rely on (unmaintained) libraries which install successfully but don’t support bdist_wheel for some reason.

They might be using setup.py install simply because they created a virtual environment with python -m venv and don’t have wheel installed.

I’m inclined to say we should make the timetable explicit. We’ve been clear on our intention for some time now that we plan on removing setup.py installs, and setuptools has publicly deprecated it, so I actually think we’ll be doing any remaining projects a favour if we give them an explicit deadline. If there’s a lot of pushback that our choice of date is too soon, we can easily change it, so being too aggressive isn’t a fatal issue.

I think if CPython wants to delay the ensurepip change to 3.12, we should also coordinate get-pip and virtualenv changes to also only not populate them if the host Python version is 3.12 or later. Otherwise users may be confused why PyPA tools behave differently from CPython tools and feels we are a bunch of stupid people not agreeing with each other.

Following the decision about feature flags, I propose --use-feature=always-install-via-wheel.

So when using --no-binary we will print a deprecation warning saying that --no-binary currently implies a deprecated call to setup.py install, and tell the user to silence the warning by using --use-feature=always-install-via-wheel.

When the transition is done, --use-feature=always-install-via-wheel becomes a no-op.

If we decide to keep the setup.py install code path for some time after the transition, then we’ll have --use-deprecated=setup-py-install.

cc/ @nlhkabu

The fact that --no-binary implies setup.py install is annoying and hard to transition. If we deprecate setup.py install we want to provide the user with a mechanism to set future-proof options and therefore silence the deprecation messages. When --no-binary (1) is used there is no way to do that, contrarily to (2) and (3) where there are solutions (respectively install wheel and fix the build error).

So I think we need to create a new, transitory option: --always-install-via-wheel or -no-use-setuppy-install. It would default to False with the goal of making it the default in the future.

Then the deprecation message can encourage users of --no-binary to activate that option and report problems they discover with that approach.

so the install is in reality just copy paste of py files… no?

No. The install also involves generating metadata, checking that metadata for dependencies and installing those, and a number of other actions.

Agreed that users having to install wheel is not ideal. But projects have to fix that by moving to pyproject.toml and/or distributing wheels. From the Python packaging tools end, we’ve publicised that, but projects move slowly. While the projects don’t change, users hit issues and the best we (pip) can do is (1) tell them how to work around the issue (install wheel) and (2) hope they put extra pressure on the projects to move forward.

It’s a problem that users don’t really know (or want/need to know) about the details of what happens with an install, and why Python code needs “building”, but I don’t know how we can solve that (without forcing end users to learn about this stuff regardless 🙁)

@xavfernandez I’m somewhat confused. The original proposal here was essentially “if we get through the build code paths, and end up at the point where we currently do setup.py install (which is when building a wheel for the project has failed), we switch to writing a deprecation warning, and then later hard error”.

Maybe I wasn’t clear with my proposal. It applies to pip installing a sdist/vcs checkout/local directory (i.e. not a wheel).

Currently:

• if pyproject.toml exists, use PEP-517/518
• otherwise, expect a setup.py:
  • if wheel is installed, run setup.py bdist_wheel and install the wheel
  • otherwise fallback to setup.py install

With what I understand from the proposal:

• if pyproject.toml exists, use PEP-517/518
• otherwise, expect a setup.py:
  • if wheel is installed, run setup.py bdist_wheel and install the wheel
  • otherwise error out and suggest to use pyproject.toml

And my suggestion:

• if pyproject.toml exists, use PEP-517/518
• otherwise, expect a setup.py:
  • if wheel is installed, run setup.py bdist_wheel and install the wheel
  • otherwise behave as if a pyproject.toml containing

     [build-system]
     requires = ["setuptools", "wheel"]
     build-backend = "setuptools.build_meta"
    

    was present

Ref: https://bugs.python.org/issue31634

Following the line of thought, maybe the warning should prompt the user to nudge package maintainers to migrate to PEP 517/518, instead of installing wheel. One way to do this would be to open a locked issue on GitHub explaining the situation, and reference the link in the message.

OTOH, our deprecation mechanism, combined with a friendly message, is a nice way to ask concerned users to report feedback in a GitHub issue. Based on that feedback we can still decide not to deprecate at the planned date or otherwise adjust the strategy.

Also, in my understanding, deprecating the setup.py install code path is “softer” that enforcing build isolation.