pip: `--require-hashes` does not correctly handle pinned package with extras
What did you want to do?
Install this requirements.txt file with pip 21.0.1 or master in a new virtual environment.
A package is pinned to a non-latest version with an extra (here, requirements[security]) and another dependency requires this package without specifying the extra.
Output
Install fails with:
...
Collecting requests[security]==2.24.0
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting six==1.15.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.25.11
Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting requests<3
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
requests<3 from https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from requests-extra==1.0.0b3->-r requirements.txt (line 116))
The resolver does not recognise that requests[security]==2.24.0 fulfils requests<3, and tries to collect latest requests. This fails as it is has no pinned hash.
Additional information
Installation succeeds with --use-deprecated=legacy-resolver.
If the requirements file has no hashes, installation succeeds with the new resolver: Collecting requests<3 resolves to latest requests (2.25.1) in the install output, but the pinned version (2.24.0) is what ends up installed.
If all packages are already installed in the environment, pip install succeeds (with Requirement already satisfied) even with the new resolver.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 36
- Comments: 57 (36 by maintainers)
Commits related to this issue
- Add sidestep for pypa/pip#9644 Should be removed before merging pypa/warehouse#8615 — committed to bskinn/warehouse by bskinn 3 years ago
- Revert "Add sidestep for pypa/pip#9644" This reverts commit c6f664a491190723034f05c1a858aa8dfa4bec8f. — committed to bskinn/warehouse by bskinn 3 years ago
- Add transitive google deps for dependabot updates workaround for pypa/pip#9644 — committed to mozilla/bigquery-etl by relud 3 years ago
- Add transitive google deps for dependabot updates workaround for pypa/pip#9644 — committed to mozilla/gcp-ingestion by relud 3 years ago
- Add transitive google deps for dependabot updates workaround for pypa/pip#9644 — committed to mozilla/lookml-generator by relud 3 years ago
- Add transitive google deps for dependabot updates (#2064) workaround for pypa/pip#9644 — committed to mozilla/bigquery-etl by relud 3 years ago
- Add transitive google deps for dependabot updates (#130) workaround for pypa/pip#9644 — committed to mozilla/lookml-generator by relud 3 years ago
- Add transitive google deps for dependabot updates (#1702) workaround for pypa/pip#9644 — committed to mozilla/gcp-ingestion by relud 3 years ago
- fix: install without dependencies * There is a dependency resolver bug with hashes & extras, see https://github.com/pypa/pip/issues/9644 * Since pip-compile already adds all dependencies this is no... — committed to DD-DeCaF/memote-webservice by Midnighter 3 years ago
- Add "--no-deps" option to pip install; workaround for https://github.com/pypa/pip/issues/9644 — committed to CivicActions/govready-q by woodt 3 years ago
- Use pip install with "--no-deps" Work-a-round for https://github.com/pypa/pip/issues/9644 Any generally a good idea, because our "requirements.txt" should always contain all needed packages ;) — committed to YunoHost-Apps/pyinventory_ynh by jedie 3 years ago
- Use pip install with "--no-deps" Work-a-round for https://github.com/pypa/pip/issues/9644 Any generally a good idea, because our "requirements.txt" should always contain all needed packages ;) — committed to YunoHost-Apps/django-for-runners_ynh by jedie 3 years ago
- Workaround for pypa/pip/issues/9644 — committed to pypi/warehouse by di 3 years ago
- Workaround for pypa/pip/issues/9644 — committed to pypi/warehouse by di 3 years ago
- Work around a bug in pip when doing a fresh install. Steps to reproduce bug: - Create a fresh virtualenv - pip install -r requirements.txt This results in: ``` ERROR: In --require-hashes mode, all ... — committed to tbeadle/CAPEv2 by deleted user 2 years ago
- Work around a bug in pip when doing a fresh install. (#717) Steps to reproduce bug: - Create a fresh virtualenv - pip install -r requirements.txt This results in: ``` ERROR: In --require-hashe... — committed to kevoreilly/CAPEv2 by tbeadle 2 years ago
- Workaround pip bug installing latest treq and not-latest twisted from hashes (#12439) The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`... — committed to matrix-org/synapse by deleted user 2 years ago
- py-pip: update to 22.0.4. 22.0.4 (2022-03-06) =================== Deprecations and Removals ------------------------- - Drop the doctype check, that presented a warning for index pages that use non... — committed to NetBSD/pkgsrc by deleted user 2 years ago
- When deploying, pass --no-deps to pip There is no reason for pip to automatically install dependencies of requested packages, because all of the dependencies have already been resolved and explicitly... — committed to lpsinger/gwcelery by lpsinger 2 years ago
- update pipt to include workaround for https://github.com/pypa/pip/issues/9644 — committed to hetida/hetida-designer by deleted user 2 years ago
One workaround for this is to use
--no-deps, which is a good idea by itself anyway (See the second warning here).That still passes, but after looking more closely I was able to figure out what will cause it to fail:
It seems like what’s happening is that
pipis not considering the child dependency with the extra to be equivalent to the child dependency without the extra, and attempting to install the latest version of the child dependency, and not using the existing pinned/hashed requirement when doing this, which causes the hash checking to fail.I made a draft PR with the failing test here: https://github.com/pypa/pip/pull/9995, but I don’t have time at the moment to continue working on it to fix the issue.
I see the problem. So your requirements are marker-guarded, and
tzdatais only pinned for platforms matching the markerI don’t know what version of Python you’re running, but chances are that version does not match this marker, so the
tzdata==2022.1pin is never considered.From your previous messages, I’m assuming this requirements.txt is generated from Poetry? That would mean this is something Poetry needs to fix, pip cannot do anything here since as far as it knows, it’s doing the right thing based on what you fed it.
If the legacy resolver also fails this is likely not the same issue as this, and also likely indicates a legistimate error.
@di I have confirmed your theory by specifying another dependency without the extras package and everything installs
FWIW #9995 did not fix the following case we encountered: one constraint with 3 extra, and another dependency constraint with 1 extra, subset of the previous 3 extra:
With pip 23.2.1:
etils[epath]andetils[enp,epath,epy]should be considered identical for the hash PoV, like the initial bug report: with vs without extra.@jedie It’s not the same. Your
flynt==0.77subdependency still requirestomlieven under Python 3.11, you should remove the conditional.@q0w @uranusjr here it is. Note that I shortened the index url since it’s a proprietary index url, which should be irrelevant in this case.
I think this might be the same issue I have encountered with pinned and hash-checking
botframework-connectorrequirements:Broken requirements file: requirements.txt
Steps to reproduce:
Actual result:
Workaround
Either install
pip <= 20.3.1, use tonybajan--use-deprecated=legacy-resolveroption or use maljub01--no-depsoption:Edit: I thought I had posted this as off-topic but after looking at the requirements.txt generated by pip-tools I see that it is specifying extras:
pyjwt[crypto]