pip: 2020 resolver does not consider hashes in the constraints file

See:

https://github.com/jwhitlock/pip-resolver-demo for a standalone demo of this issue
https://github.com/jwhitlock/pip-resolver-demo/tree/original for the original, more complex standalone demo of this issue
https://github.com/mozilla/ichnaea/pull/1289 for the full project

Update 1: The issue was not the projects multiple requirements file, but instead the combinations of using hashes and a constraints files. The current resolver uses hashes on a requirement in a constraints file, while the 2020 resolver ignores them, and fails to install because they do not have hashes. The description below describes the more complex version.

Update 2: Merged in the changes, so the default branch of pip-resolver-demo describes the simpler version, and includes the django.txt / django-versions.txt example as well.

What did you want to do?

Our requirements files include other files, as a way to only specify a requirement once for two different environments (development and building in ReadTheDocs.org):

default.txt: -c constraints.txt, -r docs.txt, -r shared.txt
docs.txt: -c constraints.txt, -r shared.txt
shared.txt: None
constraints.txt: None

All of our requirements are specified with hashes, populated with hashin.

This works, without warnings, when installing with pip install -r default.txt and pip 20.2.2.

When installing with pip 20.2.2 or pip-20.3.dev0 (today’s in-development version), this fails:

pip install -r default.txt --use-feature=2020-resolver
....
Collecting idna<3,>=2.5
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r shared.txt (line 6))

The idna==2.10 requirement is in constraints.txt, with the sha256 hash.

A similar error occurs when installing docs.txt:

pip install -r docs.txt --use-feature=2020-resolver
...
Collecting chardet<4,>=3.0.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    chardet<4,>=3.0.2 from https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 (from requests[security]==2.24.0->-r shared.txt (line 6))

Output

Here’s pip install -r default.txt --use-feature=2020-resolver:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting geoip2==4.0.2
  Downloading geoip2-4.0.2-py2.py3-none-any.whl (25 kB)
Collecting maxminddb==2.0.2
  Downloading maxminddb-2.0.2.tar.gz (285 kB)
Collecting idna<3,>=2.5
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r shared.txt (line 6))

Here’s pip install -r docs.txt --use-feature=2020-resolver:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting chardet<4,>=3.0.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    chardet<4,>=3.0.2 from https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 (from requests[security]==2.24.0->-r shared.txt (line 6))

For comparison, here’s pip install -r default.txt:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Collecting geoip2==4.0.2
  Downloading geoip2-4.0.2-py2.py3-none-any.whl (25 kB)
Collecting maxminddb==2.0.2
  Downloading maxminddb-2.0.2.tar.gz (285 kB)
Collecting idna==2.10
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet==3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting pyOpenSSL==19.1.0
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting cryptography==3.0
  Downloading cryptography-3.0-cp35-abi3-manylinux2010_x86_64.whl (2.7 MB)
Collecting packaging==20.4
  Downloading packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting Babel==2.8.0
  Downloading Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
Collecting docutils==0.15.2
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting sphinxcontrib-applehelp==1.0.2
  Downloading sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting imagesize==1.2.0
  Downloading imagesize-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting Jinja2==2.11.2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting sphinxcontrib-devhelp==1.0.2
  Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-htmlhelp==1.0.3
  Downloading sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Collecting alabaster==0.7.12
  Downloading alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting sphinxcontrib-jsmath==1.0.1
  Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting Pygments==2.6.1
  Downloading Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting snowballstemmer==2.0.0
  Downloading snowballstemmer-2.0.0-py2.py3-none-any.whl (97 kB)
Collecting sphinxcontrib-serializinghtml==1.1.4
  Downloading sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Collecting sphinxcontrib-qthelp==1.0.3
  Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting aiohttp==3.6.2
  Downloading aiohttp-3.6.2-py3-none-any.whl (441 kB)
Collecting six==1.15.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting cffi==1.14.1
  Downloading cffi-1.14.1-cp38-cp38-manylinux1_x86_64.whl (409 kB)
Collecting pyparsing==2.4.7
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pytz==2020.1
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting MarkupSafe==1.1.1
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Collecting multidict==4.7.6
  Downloading multidict-4.7.6-cp38-cp38-manylinux1_x86_64.whl (162 kB)
Collecting yarl==1.5.1
  Downloading yarl-1.5.1-cp38-cp38-manylinux1_x86_64.whl (262 kB)
Collecting async-timeout==3.0.1
  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting attrs==19.3.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting pycparser==2.20
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: MarkupSafe, maxminddb
  Building wheel for MarkupSafe (setup.py): started
  Building wheel for MarkupSafe (setup.py): finished with status 'done'
  Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-py3-none-any.whl size=12629 sha256=2f6a74f92d30e9e5c58a60f6b78a85229e279ee2229e4fced3d0c2c4e25c3d11
  Stored in directory: /root/.cache/pip/wheels/0c/61/d6/4db4f4c28254856e82305fdb1f752ed7f8482e54c384d8cb0e
  Building wheel for maxminddb (setup.py): started
  Building wheel for maxminddb (setup.py): finished with status 'done'
  Created wheel for maxminddb: filename=maxminddb-2.0.2-py3-none-any.whl size=15259 sha256=015ad8503c939cf424e3fedf475b0a3e81dd4dffbc95ed04e55235b8ce8d9f3c
  Stored in directory: /root/.cache/pip/wheels/8f/e8/68/7354267262db5ca21a7f869d544e703dcd7227b244db984b52
Successfully built MarkupSafe maxminddb
Installing collected packages: alabaster, pytz, Babel, pycparser, cffi, chardet, six, cryptography, docutils, idna, imagesize, MarkupSafe, Jinja2, Pygments, pyOpenSSL, snowballstemmer, sphinxcontrib-applehelp, sphinxcontrib-htmlhelp, sphinxcontrib-jsmath, sphinxcontrib-devhelp, sphinxcontrib-serializinghtml, pyparsing, packaging, sphinxcontrib-qthelp, attrs, multidict, yarl, async-timeout, aiohttp, certifi, urllib3, requests, Sphinx, maxminddb, geoip2
Successfully installed Babel-2.8.0 Jinja2-2.11.2 MarkupSafe-1.1.1 Pygments-2.6.1 Sphinx-3.1.2 aiohttp-3.6.2 alabaster-0.7.12 async-timeout-3.0.1 attrs-19.3.0 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 cryptography-3.0 docutils-0.15.2 geoip2-4.0.2 idna-2.10 imagesize-1.2.0 maxminddb-2.0.2 multidict-4.7.6 packaging-20.4 pyOpenSSL-19.1.0 pycparser-2.20 pyparsing-2.4.7 pytz-2020.1 requests-2.24.0 six-1.15.0 snowballstemmer-2.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.25.10 yarl-1.5.1

Additional information

Full dependency tree:

geoip2==4.0.2
  - aiohttp [required: >=3.6.2,<4.0.0, installed: 3.6.2]
    - async-timeout [required: >=3.0,<4.0, installed: 3.0.1]
    - attrs [required: >=17.3.0, installed: 19.3.0]
    - chardet [required: >=2.0,<4.0, installed: 3.0.4]
    - multidict [required: >=4.5,<5.0, installed: 4.7.6]
    - yarl [required: >=1.0,<2.0, installed: 1.5.1]
      - idna [required: >=2.0, installed: 2.10]
      - multidict [required: >=4.0, installed: 4.7.6]
  - maxminddb [required: >=2.0.0,<3.0.0, installed: 2.0.2]
  - requests [required: >=2.24.0,<3.0.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - urllib3 [required: >=1.25.2,<2.0.0, installed: 1.25.10]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.2.2]
pyOpenSSL==19.1.0
  - cryptography [required: >=2.8, installed: 3.0]
    - cffi [required: >=1.8,!=1.11.3, installed: 1.14.1]
      - pycparser [required: Any, installed: 2.20]
    - six [required: >=1.4.1, installed: 1.15.0]
  - six [required: >=1.5.2, installed: 1.15.0]
Sphinx==3.1.2
  - alabaster [required: >=0.7,<0.8, installed: 0.7.12]
  - babel [required: >=1.3, installed: 2.8.0]
    - pytz [required: >=2015.7, installed: 2020.1]
  - docutils [required: >=0.12, installed: 0.15.2]
  - imagesize [required: Any, installed: 1.2.0]
  - Jinja2 [required: >=2.3, installed: 2.11.2]
    - MarkupSafe [required: >=0.23, installed: 1.1.1]
  - packaging [required: Any, installed: 20.4]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - six [required: Any, installed: 1.15.0]
  - Pygments [required: >=2.0, installed: 2.6.1]
  - requests [required: >=2.5.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - setuptools [required: Any, installed: 49.6.0]
  - snowballstemmer [required: >=1.1, installed: 2.0.0]
  - sphinxcontrib-applehelp [required: Any, installed: 1.0.2]
  - sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
  - sphinxcontrib-htmlhelp [required: Any, installed: 1.0.3]
  - sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
  - sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
  - sphinxcontrib-serializinghtml [required: Any, installed: 1.1.4]
wheel==0.35.1

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 15 (13 by maintainers)

Links to this issue

tox-pin-deps · PyPI

Commits related to this issue

Merge #192 192: Update pip to 20.2.4 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.3** to **20.2.4**. <details> <summary>Changelog</summary> ... — committed to duckinator/emanate by bors[bot] 4 years ago
Explain resolver changes affecting constraints files Related to #8792, #8076, #9020. — committed to brainwane/pip by brainwane 4 years ago
Explain resolver changes affecting constraints files Related to #8792, #8076, #9020. — committed to brainwane/pip by brainwane 4 years ago
Explain resolver changes affecting constraints files Related to #8792, #8076, #9020. — committed to brainwane/pip by brainwane 4 years ago
Explain resolver changes affecting constraints files Related to #8792, #8076, #9020. — committed to brainwane/pip by brainwane 4 years ago
Explain resolver changes affecting constraints files Related to #8792, #8076, #9020. — committed to brainwane/pip by brainwane 4 years ago
Explain resolver changes affecting constraints files Related to #8792, #8076, #9020. — committed to brainwane/pip by brainwane 4 years ago
Merge #195 195: Update pip to 20.3 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**. <details> <summary>Changelog</summary> ### ... — committed to duckinator/emanate by bors[bot] 4 years ago
Merge #195 195: Update pip to 20.3 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**. <details> <summary>Changelog</summary> ### ... — committed to duckinator/emanate by bors[bot] 4 years ago
Merge #195 195: Update pip to 20.3 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**. <details> <summary>Changelog</summary> ### ... — committed to duckinator/emanate by bors[bot] 4 years ago
Merge #194 #197 194: Update pytest-pylint to 0.18.0 r=duckinator a=pyup-bot This PR updates [pytest-pylint](https://pypi.org/project/pytest-pylint) from **0.17.0** to **0.18.0**. *The bot wasn't... — committed to duckinator/emanate by bors[bot] 4 years ago

Most upvoted comments

@jwhitlock I think the fix to the issue is surfacing another different bug (that was not triggered previously since pip fails before hitting it). It would be best to track it in another issue.

uranusjr on Oct 20, 2020

The merging is done in the Hashes class (which is on InstallRequirement). I traced the code and believe it’s possible to only change the new resolver implementation to do the intersection logic (and keep the legacy resolver as-is, performing union). PR coming shortly.

uranusjr on Sep 3, 2020

tl;dr; I feel that we should be cautious, but I don’t have a strong objection.

I wouldn’t describe the new resolver changes as a “minor redesign” - I think we did a fairly comprehensive overhaul and stripped constraints files down to being purely a way to specify global (version) limits for packages. That gave us a very specific behaviour, that translates well to how pip decides which version to install (don’t even consider versions that don’t match the limits in the constraint file).

I’m fine with adding extra functionality to constraint files that is in the same vein - removing certain candidates from consideration up front. As @uranusjr says, hashes can be viewed in this way, and so I feel that it’s a reasonable extension. However, that logic does mean that IMO the only reasonable interpretation of the case when hashes are in both constraints and requirements is “intersection” - pip never considers anything that doesn’t match the constraints, so if a requirement has a conflicting hash, pip will see nothing and hence will fail. I’d be unhappy with “requirements override” or “union” because they don’t fit that model.

My mental model of a constraints file is that it is the requirements of the requirements that I care about.

That’s not how I viewed them when I designed the new implementation. As I said above, the intended interpretation is that constraints limit what candidates pip sees. Most of the time, I don’t expect there to be much difference in the viewpoints (which has positive and negative aspects - we won’t conflict with people’s expectations, which is good, but people may want to push constraints in a direction we didn’t intend, which is less so…)

pfmoore on Aug 30, 2020