pip-tools: pip-compile fails with pip==22.0

this is a big bug 🤯

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella.

Nope. The fundamental problem here is that pip does not provide any public API – see https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program. pip-tools is one of the few projects in the wild that still uses pip’s internal code as if it’s an API and this sort of breakage is exactly what the risk of that usage pattern is.

Just to add my 2 cents, I wish pip would add a compile and sync flag to effectively integrate pip-tools’s functionality into standard pip. Then issues like this one here won’t arise in stable pip releases and we would have good dependency management out of the box. Probably not going to happen, but one can dream 😃

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella.

Nope. The fundamental problem here is that pip does not provide any public API – see https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program. pip-tools is one of the few projects in the wild that still uses pip’s internal code as if it’s an API and this sort of breakage is exactly what the risk of that usage pattern is.

In fact, Jazzband has more “human resources” than the PyPA does. I’m a member of both, and I can tell you that there’s like… ~100x more people in the Jazzband org. 😃

I’d suggest a closer eye is paid to the integration of pip with pip-tools, especially in the lead up to pip’s quarterly releases.

It effectively comes down to pip-tools maintainers having enough resources and time to actually work on keeping up with pip’s changes. This comes down to someone actually doing things – and, well, evidently, no one has had the time/energy to do that.

I’ll note that this project is a part of Jazzband – realistically, basically anyone who is reading this comment can pick up the responsibility to do the work necessary to bring this project up to date. From https://jazzband.co/: “We are all part of this”.

You can also contribute to the whole Jazzband org with $$$ as well, which I’ll encourage you to do if you’re deriving monetary value from this project – like if you’re running a commercial operation that uses thi tool.

This repo does have a daily cron to test against pip main, which is really good.

https://github.com/jazzband/pip-tools/actions/workflows/cron.yml

However, the Python 3.6 job started failing three months ago when pip dropped support for 3.6:

Which masked all the jobs started failing two months ago due to other internal pip changes:

https://github.com/jazzband/pip-tools/runs/4195776918?check_suite_focus=true

https://github.com/jazzband/pip-tools/runs/4419708428?check_suite_focus=true

---

I’m not sure whether a change in organisation would necessarily help, the pip team will continue to change their private internal functions as they need to (and there will be more changes to use Rich more for better output).

Wherever it lives, I’d suggest a closer eye is paid to the integration of pip with pip-tools, especially in the lead up to pip’s quarterly releases. (And look out for the pip release issues for planned release dates e.g. https://github.com/pypa/pip/issues/10726.)

pip is not doing SemVer; <23 would pin updates to 2022. The only way to prevent this happening again is #1560 & diligently keeping the CI green.

Also as a reminder: what broke here is strictly an implementation detail. No interpretation of SemVer ever would bump their major version over this.

The workaround, for folks affected, is pip install pip<22 in the environment where you’re using pip-tools.

A similar occurrence happened when pip 21.3 was released last year: https://github.com/jazzband/pip-tools/issues/1503.

On it there was some mention of making sure the pip-tools and pip maintainers stay in sync when breaking changes are made. Since it seems that pip is pretty liberal about refactoring their internals (which, to be clear, is their prerogative since it is internal) so that extra communication with pip-tools could help.

I wonder if this could be alleviated by moving the pip-tools project under the pypa umbrella. One of the pypa’s most important products uses pip-tools so I would imagine they would see value in ensuring it stays in sync with pip. (As a side note, the fact that warehouse uses pip-tools and not pipenv is interesting, to say the least, considering that pipenv is the officially recommended tool by the pypa…). This suggestion is mirroring the sentiment here.

@RAbraham Or as commands, you need to either install pip-tools first and then downgrade pip, e.g. pip install pip-tools; pip install 'pip<22'. Or just provide the limitation in the same command pip install pip-tools 'pip<22'. But yes, it probably makes more sense to put the limitations into your dependency spec file.

Please see https://github.com/jazzband/pip-tools/pull/1567 for a fix.

This fixed my build issues. Thanks for all your efforts everyone! Look forward to the patch.

        python -m pip install --upgrade pip~=21.3
        pip install --upgrade pip-tools

I pushed a merge request that fixes the issue for me, but it’s failing tests.

https://github.com/jazzband/pip-tools/pull/1559

Hey, as a quickfix I’d suggest to simply change the assertion to cover pip._internal.utils.logging.RichPipStreamHandler as well as the currently existing to have it running. This would decouple the current issue of people who want to have it working again from the discussion of pinning pip or how to use the internals. Feel free to leave comments at https://github.com/jazzband/pip-tools/pull/1565 if you like or dislike that idea.

I think we may have to do more than pip install pip<22? I just tried that but my next command to install pip-tools downloads pip 22 again?

pip install "pip<22"


pip install pip-tools


Collecting pip>=21.2
  Downloading pip-22.0.2-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from pip-tools) (57.0.0)
Collecting click>=7
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Requirement already satisfied: wheel in /usr/local/lib/python3.7/site-packages (from pip-tools) (0.36.2)
Collecting pep517
  Downloading pep517-0.12.0-py2.py3-none-any.whl (19 kB)
Collecting importlib-metadata
  Using cached importlib_metadata-4.10.1-py3-none-any.whl (17 kB)
Collecting typing-extensions>=3.6.4
  Using cached typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Collecting zipp>=0.5
  Using cached zipp-3.7.0-py3-none-any.whl (5.3 kB)
Collecting tomli>=1.1.0
  Downloading tomli-2.0.0-py3-none-any.whl (12 kB)
Installing collected packages: zipp, typing-extensions, tomli, importlib-metadata, pip, pep517, click, pip-tools
  Attempting uninstall: pip
    Found existing installation: pip 21.1.3
    Uninstalling pip-21.1.3:
      Successfully uninstalled pip-21.1.3
Successfully installed click-8.0.3 importlib-metadata-4.10.1 pep517-0.12.0 pip-22.0.2 pip-tools-6.4.0 tomli-2.0.0 typing-extensions-4.0.1 zipp-3.7.0

https://github.com/pypa/pip/issues/10726#issuecomment-1025206695:

That looks like a bug on their end – they’re using pip’s internals (something that is not supported) and quite possibly they weren’t prepared for https://github.com/pypa/pip/issues/10461. 😃