pip-tools: pip-compile doesn't support the new pip resolver

#1539 adds support for new resolver and fixes the issue:

❯ pip-compile --resolver backtracking
#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile --resolver=backtracking
#
boto3==1.21.4
    # via -r requirements.in
botocore==1.24.5
    # via
    #   boto3
    #   s3transfer
jmespath==0.10.0
    # via
    #   boto3
    #   botocore
python-dateutil==2.8.2
    # via botocore
s3transfer==0.5.1
    # via boto3
six==1.16.0
    # via python-dateutil
urllib3==1.26.8
    # via botocore

The resolver is released as part of pip-tools v6.8.0 🎉

@atugushev I can’t say how much this saved the day for me - thank you for getting this in there when you did!

confirmed pip-compile can’t handle,

boto3
botocore==1.20.20

but pip finds valid solution:

boto3==1.17.20
botocore==1.20.20

So sounds like no ones working on this?

This issue is critical as the new resolver is days away from becoming default. If you was one of those that added use-feature = 2020-resolver to your ~/.config/pip/pip.conf in order to benefit from extra conflict resolution in newer resolver you are unable to use pip-compile at all.

Well pip never considers the internal API stable right? 😃

For my purposes I don’t actually need the new resolver logic except to deal with https://github.com/pypa/pip/issues/5780. If you don’t think the new resolver internals are stable enough for pip-tools yet, I’m sure I can come up with some hack to work around this issue. 😃

@Eric-Arellano there’s a new lock file spec here too https://www.python.org/dev/peps/pep-0665/

Hi, I’m a maintainer at https://github.com/pantsbuild/pants. We’re working on a revamp of our 3rd-party support with lockfiles at the crux of the changes. We’ve been using pip-compile so far and are really happy with it so far, great UX!

This issue is a blocker for us to call the project complete. We’re interested in taking on this issue, but are trying to scope it out: how much work would you anticipate this change will take?

Also, would anyone be available to help, like giving some tips or pair programming?

Finally, Pants still supports Python 2.7, so we’d ideally backport this feature to pip-tools 5, in addition to pip-tools 6. How feasible do you expect a backport like that would be?

I have simplest example: req.txt:

asyncpg ~= 0.15.0
asyncpgsa >= 0.22.0

pip-compile fails with:

Could not find a version that matches asyncpg>=0.22.0,~=0.15.0 (from -r req.txt (line 1))
Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.1, 0.6.3, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.20.0, 0.20.0, 0.20.1, 0.20.1, 0.21.0, 0.21.0, 0.22.0, 0.22.0
There are incompatible versions in the resolved dependencies:
  asyncpg~=0.15.0 (from -r req.txt (line 1))
  asyncpg>=0.22.0 (from asyncpgsa==0.27.1->-r req.txt (line 2))

but pip install succeed with:

...
Successfully installed asyncpg-0.15.0 asyncpgsa-0.27.0 greenlet-1.0.0 sqlalchemy-1.4.6

Tested on both pypi and github-master versions of pip-tools. Latest pip (since pip-tools required it).

If I change req.txt as:

asyncpg ~= 0.15.0
asyncpgsa >= 0.22.0, < 0.27.1

Then I get pip-compile succeed with:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile req.txt
#
asyncpg==0.15.0
    # via
    #   -r req.txt
    #   asyncpgsa
asyncpgsa==0.27.0
    # via -r req.txt
greenlet==1.0.0
    # via sqlalchemy
sqlalchemy==1.4.6
    # via asyncpgsa

I…uh…played a bit with the settings hence and forth and suddenly it started working and I can’t break it anymore. 😳 It was 100% 5.4.0/20.3 before. I guess my computer is just haunted.

apologies, that was 5.1.2, works with 5.3.1 👍 @atugushev