pip: PEP 668 support breaks --user/--editable

Description

@obfusk filed Debian bug #1030335 about the implementation of PEP 668 support.

I think it’s best if this is discussed here, with upstream involvement. cc @geofft @doko42 @FFY00 @dstufft @uranusjr @ehashman @pradyunsg @kitterma who were involved in the PEP or pip in Debian.

Hi!

I just updated python3-pip and was greeted with a NEWS message about PEP 668 support:

This version of pip introduces PEP 668 support. Debian’s python3.11 interpreter will soon (>= 3.11.1-3) declare the installation to be EXTERNALLY-MANAGED, instructing pip to disallow package installation outside virtualenvs.

See: https://peps.python.org/pep-0668/

Practically, this means that you can’t use pip to install packages outside a virtualenv, on a Debian system, any more.

See /usr/share/doc/python3.11/README.venv for more details. If that isn’t available yet, check: https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

Not being able to install packages system-wide seems like a good idea, I fully support that.

But I often install the Python packages I’m developing under my own user account with “pip install -e”, which is also made impossible by this change.

And I intentionally do not use virtualenvs for this because I want to have all dependencies provided by Debian packages, not downloaded from PyPI.

And as far as I can tell there is not even an option I can give to pip to tell it to allow me to install to ~/.local anyway. PEP 668 mentions a --break-system-packages (as an example), but such an option doesn’t seem to actually exist.

FC

This bug is about a Debian-user-specific issue (not wanting to depend on PyPI), but I don’t think this is something Debian should try to solve, alone.

When the PEP was drafted, we were assuming there’d be some sort of mechanism to override it (allow the user to break their system and keep both pieces), I quote:

The installer should have a way for the user to override these rules, such as a command-line flag --break-system-packages. This option should not be enabled by default and should carry some connotation that its use is risky.

That didn’t make it into the implementation, so I suggested deleting the EXTERNALLY-MANAGED file in the documentation Debian is providing about this. It’s not great, but it’s about the only suggestion I can make:

https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

Does pip have any thoughts on providing an override mechanism?

There is some value in not having an override (we don’t need to keep testing that all these schemes work, for one). But as we transition, we’re going to be breaking use-cases…

Expected behavior

No response

pip version

23.0

Python version

3.11.1

OS

Debian GNU/Linux

How to Reproduce

apt install python3-pip
check out a python source package.
pip install -e .

Output

No response

Code of Conduct

I agree to follow the PSF Code of Conduct.

About this issue

Original URL
State: closed
Created a year ago
Comments: 19 (7 by maintainers)

Commits related to this issue

try --break-system-packages Ref: https://github.com/pypa/pip/issues/11776 Ref: https://peps.python.org/pep-0668/ Ref: https://bugs.gentoo.org/895410 — committed to curl/curl-for-win by vszakats a year ago
try --break-system-packages Ref: https://github.com/pypa/pip/issues/11776 Ref: https://github.com/pypa/pip/pull/11780 Ref: https://peps.python.org/pep-0668/ Ref: https://bugs.gentoo.org/895410 — committed to curl/curl-for-win by vszakats a year ago
python: install pefile as system package on debian/msys2 If losing all installed Python (pip) packages on every .x upgrade was not enough (on macOS/Homebrew at least), now it became impossible to use... — committed to curl/curl-for-win by vszakats a year ago

Most upvoted comments

Deleting /usr/lib/pypy3.9/EXTERNALLY-MANAGED fixes my problems with pip and pypy3 on Debian.

futzu on Feb 13, 2023

I do not want to install into system site-packages, but I do want to continue to keep using --user (since a venv doesn’t work for my use case), so I added this to my ~/.config/pip/pip.conf:

[global]
break-system-packages = true

obfusk on Feb 17, 2023

Note that while “practicality beats purity” and all that, and we don’t yet have the override flag released, I feel it’s important to note that deleting EXTERNALLY-MANAGED is not the intended way of overriding this protection, and should be treated as a temporary hack to work around the current lack of an override flag.

If you want to install into the system site-packages (and understand the risks[^1]), I would recommend that once the next version of pip is released with the override flag, you either use the --break-system-packages flag while invoking pip, or you set the PIP_BREAK_SYSTEM_PACKAGES environment variable just for the duration of your build (don’t leave it set in the resulting runtime container!).

[^1]: Of course, we can’t dictate how people choose to build their containers, and we do understand that the container environment is atypical in many ways, but please understand the risks before overriding things.

pfmoore on Feb 15, 2023

Disallowing --user is intentional

According to the PEP, so is having an override mechanism. Which there isn’t. So the current unconditional breakage of --user is not in accordance with the PEP.

Using a virtual environment with --system-site-packages is a much better solution.

Why? I can understand requiring some kind of “I know what I’m doing” option (as the PEP mentions), after all --system-site-packages is also opt-in. But why is is better to be forced to use a virtual environment when I always want that to be active?

obfusk on Feb 5, 2023

FWIW, I can use e.g. python3 -mvenv --system-site-packages --without-pip ~/.venv as a workaround (and even automatically activate that venv in ~/.bashrc).

But that’s more work, and not compatible with what’s already installed in ~/.local.

I don’t see how that is any better than continuing to allow --user.

I’m still using the system site packages like before, so I don’t think it “breaks the system” any less than --user does.

obfusk on Feb 3, 2023