pip: [regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins
What did you want to do?
Use pip-tools generated requirements.txt + constraints.txt pair with hashes via pip install -r requirements.txt -c constraints.txt
. It seems to be unintentionally broken with the new resolvelib-based dependency resolver and works with the old one.
This seems to be related to https://github.com/pypa/pip/issues/8792 but PR https://github.com/pypa/pip/pull/8839 that claims to fix it doesn’t add the specific case I’m presenting here to tests and so it is probably overlooked.
Please find the repro below. Note that I’m adding an unconstrainted direct env dep to myst.in
and generate the concrete pins with hashes in myst.txt
for simplicity but the case I’ve actually hit in the wild was with >= 0.10.0
(that shouldn’t matter here).
STR + Output
$ python -m pip install pip-tools -U
Collecting pip-tools
Downloading pip_tools-5.4.0-py2.py3-none-any.whl (45 kB)
|████████████████████████████████| 45 kB 1.3 MB/s
Requirement already satisfied, skipping upgrade: six in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (1.15.0)
Requirement already satisfied, skipping upgrade: click>=7 in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (7.1.2)
Requirement already satisfied, skipping upgrade: pip>=20.1 in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (20.2.3)
Installing collected packages: pip-tools
Attempting uninstall: pip-tools
Found existing installation: pip-tools 5.3.1
Uninstalling pip-tools-5.3.1:
Successfully uninstalled pip-tools-5.3.1
Successfully installed pip-tools-5.4.0
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.
$ echo myst-parser > myst.in
$ python -m piptools compile --generate-hashes --output-file myst.txt myst.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes --output-file=myst.txt myst.in
#
alabaster==0.7.12 \
--hash=sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359 \
--hash=sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02 \
# via sphinx
attrs==20.3.0 \
--hash=sha256:31b2eced602aa8423c2aea9c76a724617ed67cf9513173fd3a4f03e3a929c7e6 \
--hash=sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700 \
# via markdown-it-py
babel==2.9.0 \
--hash=sha256:9d35c22fcc79893c3ecc85ac4a56cde1ecf3f19c540bba0922308a6c06ca6fa5 \
--hash=sha256:da031ab54472314f210b0adcff1588ee5d1d1d0ba4dbd07b94dba82bde791e05 \
# via sphinx
certifi==2020.12.5 \
--hash=sha256:1a4995114262bffbc2413b159f2a1a480c969de6e6eb13ee966d470af86af59c \
--hash=sha256:719a74fb9e33b9bd44cc7f3a8d94bc35e4049deebe19ba7d8e108280cfd59830 \
# via requests
chardet==3.0.4 \
--hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
--hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
# via requests
docutils==0.16 \
--hash=sha256:0c5b78adfbf7762415433f5515cd5c9e762339e23369dbe8000d84a4bf4ab3af \
--hash=sha256:c2de3a60e9e7d07be26b7f2b00ca0309c207e06c100f9cc2a94931fc75a478fc \
# via myst-parser, sphinx
idna==2.10 \
--hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6 \
--hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 \
# via requests
imagesize==1.2.0 \
--hash=sha256:6965f19a6a2039c7d48bca7dba2473069ff854c36ae6f19d2cde309d998228a1 \
--hash=sha256:b1f6b5a4eab1f73479a50fb79fcf729514a900c341d8503d62a62dbc4127a2b1 \
# via sphinx
jinja2==2.11.2 \
--hash=sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0 \
--hash=sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035 \
# via sphinx
markdown-it-py==0.5.6 \
--hash=sha256:6143d11221495edbf71beb7e455821ae6c8f0156710a1b11812662ed6dbd165b \
--hash=sha256:dcfe4a0c6bef711cb6c42494ebf23a3cfe6f249bf995556498497dd8193bfc22 \
# via myst-parser
markupsafe==1.1.1 \
--hash=sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473 \
--hash=sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161 \
--hash=sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235 \
--hash=sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5 \
--hash=sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42 \
--hash=sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff \
--hash=sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b \
--hash=sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1 \
--hash=sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e \
--hash=sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183 \
--hash=sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66 \
--hash=sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b \
--hash=sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1 \
--hash=sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15 \
--hash=sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1 \
--hash=sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e \
--hash=sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b \
--hash=sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905 \
--hash=sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735 \
--hash=sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d \
--hash=sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e \
--hash=sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d \
--hash=sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c \
--hash=sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21 \
--hash=sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2 \
--hash=sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5 \
--hash=sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b \
--hash=sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6 \
--hash=sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f \
--hash=sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f \
--hash=sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2 \
--hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
--hash=sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be \
# via jinja2
myst-parser==0.12.10 \
--hash=sha256:4612c46196e0344bb7e49dbc3deb288f9b9a88fcf6e9f210f7f3ea5bc9899bfc \
--hash=sha256:a5311da4398869e596250d5a93b523735c3beb8bc9d3eba853223c705802043b \
# via -r myst.in
packaging==20.7 \
--hash=sha256:05af3bb85d320377db281cf254ab050e1a7ebcbf5410685a9a407e18a1f81236 \
--hash=sha256:eb41423378682dadb7166144a4926e443093863024de508ca5c9737d6bc08376 \
# via sphinx
pygments==2.7.3 \
--hash=sha256:ccf3acacf3782cbed4a989426012f1c535c9a90d3a7fc3f16d231b9372d2b716 \
--hash=sha256:f275b6c0909e5dafd2d6269a656aa90fa58ebf4a74f8fcf9053195d226b24a08 \
# via sphinx
pyparsing==2.4.7 \
--hash=sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1 \
--hash=sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b \
# via packaging
pytz==2020.4 \
--hash=sha256:3e6b7dd2d1e0a59084bcee14a17af60c5c562cdc16d828e8eba2e683d3a7e268 \
--hash=sha256:5c55e189b682d420be27c6995ba6edce0c0a77dd67bfbe2ae6607134d5851ffd \
# via babel
pyyaml==5.3.1 \
--hash=sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97 \
--hash=sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76 \
--hash=sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2 \
--hash=sha256:6034f55dab5fea9e53f436aa68fa3ace2634918e8b5994d82f3621c04ff5ed2e \
--hash=sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648 \
--hash=sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf \
--hash=sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f \
--hash=sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2 \
--hash=sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee \
--hash=sha256:ad9c67312c84def58f3c04504727ca879cb0013b2517c85a9a253f0cb6380c0a \
--hash=sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d \
--hash=sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c \
--hash=sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a \
# via myst-parser
requests==2.25.0 \
--hash=sha256:7f1a0b932f4a60a1a65caa4263921bb7d9ee911957e0ae4a23a6dd08185ad5f8 \
--hash=sha256:e786fa28d8c9154e6a4de5d46a1d921b8749f8b74e28bde23768e5e16eece998 \
# via sphinx
snowballstemmer==2.0.0 \
--hash=sha256:209f257d7533fdb3cb73bdbd24f436239ca3b2fa67d56f6ff88e86be08cc5ef0 \
--hash=sha256:df3bac3df4c2c01363f3dd2cfa78cce2840a79b9f1c2d2de9ce8d31683992f52 \
# via sphinx
sphinx==3.3.1 \
--hash=sha256:1e8d592225447104d1172be415bc2972bd1357e3e12fdc76edf2261105db4300 \
--hash=sha256:d4e59ad4ea55efbb3c05cde3bfc83bfc14f0c95aa95c3d75346fcce186a47960 \
# via myst-parser
sphinxcontrib-applehelp==1.0.2 \
--hash=sha256:806111e5e962be97c29ec4c1e7fe277bfd19e9652fb1a4392105b43e01af885a \
--hash=sha256:a072735ec80e7675e3f432fcae8610ecf509c5f1869d17e2eecff44389cdbc58 \
# via sphinx
sphinxcontrib-devhelp==1.0.2 \
--hash=sha256:8165223f9a335cc1af7ffe1ed31d2871f325254c0423bc0c4c7cd1c1e4734a2e \
--hash=sha256:ff7f1afa7b9642e7060379360a67e9c41e8f3121f2ce9164266f61b9f4b338e4 \
# via sphinx
sphinxcontrib-htmlhelp==1.0.3 \
--hash=sha256:3c0bc24a2c41e340ac37c85ced6dafc879ab485c095b1d65d2461ac2f7cca86f \
--hash=sha256:e8f5bb7e31b2dbb25b9cc435c8ab7a79787ebf7f906155729338f3156d93659b \
# via sphinx
sphinxcontrib-jsmath==1.0.1 \
--hash=sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178 \
--hash=sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8 \
# via sphinx
sphinxcontrib-qthelp==1.0.3 \
--hash=sha256:4c33767ee058b70dba89a6fc5c1892c0d57a54be67ddd3e7875a18d14cba5a72 \
--hash=sha256:bd9fc24bcb748a8d51fd4ecaade681350aa63009a347a8c14e637895444dfab6 \
# via sphinx
sphinxcontrib-serializinghtml==1.1.4 \
--hash=sha256:eaa0eccc86e982a9b939b2b82d12cc5d013385ba5eadcc7e4fed23f4405f77bc \
--hash=sha256:f242a81d423f59617a8e5cf16f5d4d74e28ee9a66f9e5b637a18082991db5a9a \
# via sphinx
urllib3==1.26.2 \
--hash=sha256:19188f96923873c92ccb987120ec4acaa12f0461fa9ce5d3d0772bc965a39e08 \
--hash=sha256:d8ff90d979214d7b4f8ce956e80f4028fc6860e4431f731ea4a8c08f23f99473 \
# via requests
# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools
The generated requirements file may be rejected by pip install. See # WARNING lines for details.
$ python -m pip -V
pip 20.2.3 from ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pip (python 3.9)
$ python -m pip --use-feature=2020-resolver install -r myst.in -c myst.txt
Collecting myst-parser
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
myst-parser from https://files.pythonhosted.org/packages/2f/f0/bf66b8c5428dd7352cbc258981dce95fcd3ae63e66aa843f3b0f5986c3ba/myst_parser-0.12.10-py3-none-any.whl#sha256=a5311da4398869e596250d5a93b523735c3beb8bc9d3eba853223c705802043b (from -r myst.in (line 1))
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.
Additional information
Example crash: https://github.com/ansible/pylibssh/pull/157/checks?check_run_id=1511500384#step:9:15.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 8
- Comments: 15 (14 by maintainers)
Commits related to this issue
- Use `-r` instead of `-c` paradigm @ tox build-docs Ref: https://github.com/pypa/pip/issues/9243 — committed to ansible/pylibssh by webknjaz 4 years ago
- Use `-r` instead of `-c` because of pip bug 9243 Ref: https://github.com/pypa/pip/issues/9243 — committed to webknjaz/ansible-galaxy-collection-resolver by webknjaz 3 years ago
- Use `-r` in `make-changelog` tox env This is necessary because the most recent versions of pip have a bug with using `-c` having `--require-hashes` or entries with extras. Ref: https://github.com/py... — committed to ansible/pylibssh by webknjaz 3 years ago
- Bump pip from 22.0.4 to 22.1 (PR #4238) Bumps [pip](https://github.com/pypa/pip) from 22.0.4 to 22.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/b... — committed to inmanta/inmanta-core by dependabot[bot] 2 years ago
- Bump pip from 22.0.4 to 22.1 (PR #4237) Bumps [pip](https://github.com/pypa/pip) from 22.0.4 to 22.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/b... — committed to inmanta/inmanta-core by dependabot[bot] 2 years ago
- py-pip: updated to 22.1.2 22.1.2 (2022-05-31) =================== Bug Fixes --------- - Revert <https://github.com/pypa/pip/issues/10979> since it introduced a regression in certain edge cases. - Fi... — committed to NetBSD/pkgsrc by deleted user 2 years ago
- Stop pinning hashes in `requirements-build.txt` At least, until https://github.com/pypa/pip/issues/9243 is fixed. — committed to webknjaz/pylibssh by webknjaz 2 years ago
- Merge branch 'reproducibility/pip-tools-pep517-build-constraints' into devel This patch attempts to integrate a forgotten constraints file into the tox setup. The idea is to make PEP 517 builds repro... — committed to ansible/pylibssh by webknjaz 2 years ago
I’m on board for fixing the logic in the hash checking, to check that exactly 1 candidate is available, instead of checking that the requirement it got is a pin.
Beyond that, I don’t have any thoughts.
I’d be happy to help if someone wants to work on a pull request for this.
I’ve quickly looked into this, and found that
_iter_candidates_from_constraints
is likely/maybe meant to help with that, but does not yield a candidate, since the constraint has nolinks
.Code ref: https://github.com/pypa/pip/blob/9d25d2aba53928cb77149b8854b6b8a8ccfd81b8/src/pip/_internal/resolution/resolvelib/factory.py#L324-L345
Additional info / clarifying my use case
My use case is having a
requirements.txt
file with hashes (produced by pdm, pip-tools/pip-compile etc), and then using this for constraints to install some dependency out of it:req.txt
:pip install -c req.txt pdm
then fails withERROR: In --require-hashes mode, all requirements must have their versions pinned with ==.
(apparently having hashes in a constraints file enables require-hashes mode).With
pip install -c req.txt pdm==1.9.0
(adding/copying the version from the constraint):ERROR: Hashes are required in --require-hashes mode, but they are missing from some requirements. … pdm==1.9.0 --hash=sha256:caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb
).Previous discussion: #9020