pip: pip 20.2.4 with 2020 resolver does not use pinned version in the constraints file

What did you want to do?

Consider this requirements.txt:

-c constraints.txt

Django==3.1 \
    --hash=sha256:1a63f5bb6ff4d7c42f62a519edc2adbb37f9b78068a5a862beff858b68e3dc8b \
    --hash=sha256:2d390268a13c655c97e0e2ede9d117007996db692c1bb93eabebd4fb7ea7012b

Django 3.1 has three dependencies, listed with hashes in constraints.txt:

pytz==2020.1 \
    --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \
    --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048
sqlparse==0.3.1 \
    --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \
    --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548
asgiref==3.2.10 \
    --hash=sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a \
    --hash=sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed

With pip 20.2.4, this installs the four packages:

pip install -r requirements.txt

Using the 2020 resolver fails:

pip install -r requirements.txt --use-feature=2020-resolver

It also fails with the in-development version of pip.

Output

Working output:

$ pip install -r requirements.txt 
Collecting Django==3.1
  Using cached Django-3.1-py3-none-any.whl (7.8 MB)
Collecting asgiref==3.2.10
  Using cached asgiref-3.2.10-py3-none-any.whl (19 kB)
Collecting sqlparse==0.3.1
  Using cached sqlparse-0.3.1-py2.py3-none-any.whl (40 kB)
Collecting pytz==2020.1
  Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Installing collected packages: pytz, sqlparse, asgiref, Django
Successfully installed Django-3.1 asgiref-3.2.10 pytz-2020.1 sqlparse-0.3.1

Failed output:

$ pip install -r requirements.txt --use-feature=2020-resolver
Collecting Django==3.1
  Using cached Django-3.1-py3-none-any.whl (7.8 MB)
Collecting asgiref~=3.2.10
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    asgiref~=3.2.10 from https://files.pythonhosted.org/packages/d5/eb/64725b25f991010307fd18a9e0c1f0e6dff2f03622fc4bcbcdb2244f60d6/asgiref-3.2.10-py3-none-any.whl#sha256=9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed (from Django==3.1->-r requirements.txt (line 3))

The dependency that causes the error can vary from call to call.

Additional information

Here’s the output of pipdeptree for the successful case:

$ pipdeptree
Django==3.1
  - asgiref [required: ~=3.2.10, installed: 3.2.10]
  - pytz [required: Any, installed: 2020.1]
  - sqlparse [required: >=0.2.2, installed: 0.3.1]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.2.4]
setuptools==49.6.0
wheel==0.35.1

This was first reported in issue #8792, which is marked as fixed in pip 20.2.4 with PR #8839.

A working demo is at https://github.com/jwhitlock/pip-resolver-demo. I’ve simplified it to this example since it was first written.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Got here from #9243. It seems weird to suddenly expect that, when a constraints file specifies a version pin and a hash, the resolver would ignore the hash completely when requirements doesn’t specify it. That seems to make constraints files essentially useless, because now for anyone using require-hashes, it makes no sense to specify the dependency in both requirements.txt and constraints.txt, and it also doesn’t make sense to go through repeated installs and find every subdep specified with a range and special-casing those for requirements.txt. Is there some reasoning someone can point to for breaking this flow?

+2
aawilson on Jan 6, 2021

I modified the title to reflect my observation to the issue. The error message indicates that hashes are not the issue, but the resolver not “using” the asgiref==3.2.10 line in constraints. It uses asgiref~=3.2.10 from Django instead, causing the failure.

With the current implementation, the user is expected to == all requirements as concrete requirements, and constraints don’t count. Personally I feel both pip’s current behaviour and your expectation make sense, and is (very) slightly in favour of making this work as you expect.

+1
uranusjr on Oct 21, 2020

They are basically a set of version specifications like

foo==1.0
bar>=2.0

The difference isn’t so much in how they are specified (except that the new resolver doesn’t accept more complicated requirement syntax like extras or markers in constraint files) as in how they are used. In the old resolver, they were essentially treated as another set of requirements, but they didn’t trigger an install unless there was a “non-constraint” requirement for that package as well. In the new resolver, constraints are applied much earlier, as part of finding candidates to feed into the resolver, so that the resolver never even sees candidates that don’t match the constraints (which IMO is much closer in behaviour to what I think “constraint” means). The new resolver also doesn’t see the constraints, as they have been applied before the resolver gets involved.

The consequence here is that the resolver never sees a requirement asgiref==3.2.10 because that’s in a constraint file. But that’s OK, as it never sees a version of asgiref other than 3.2.10, because of the constraint, so there’s no need to have it as a requirement as well.

The problem is that hash-checking mode requires that all requirements are specified as a == match on a version, and the only requirement you have is asgiref~=3.2.10, presumably from a dependency somewhere. So hash-checking mode complains that it’s not an == match. If you add asgiref==3.2.10 to your command line (or to a requirements file), that would act as a version pin and satisfy hash-checking mode.

+1
pfmoore on Oct 20, 2020
Read more comments on GitHub
← pip: Pip 20.2.4 goes into infinite resolution of dependencies
pip: Pip 20.3+ break proxy connection →