pip: [regression] resolvelib-based dependency processing of constraints with hashes fails to "see" the exact pins

What did you want to do? Use pip-tools generated requirements.txt + constraints.txt pair with hashes via pip install -r requirements.txt -c constraints.txt. It seems to be unintentionally broken with the new resolvelib-based dependency resolver and works with the old one.

This seems to be related to https://github.com/pypa/pip/issues/8792 but PR https://github.com/pypa/pip/pull/8839 that claims to fix it doesn’t add the specific case I’m presenting here to tests and so it is probably overlooked.

Please find the repro below. Note that I’m adding an unconstrainted direct env dep to myst.in and generate the concrete pins with hashes in myst.txt for simplicity but the case I’ve actually hit in the wild was with >= 0.10.0 (that shouldn’t matter here).

STR + Output

$ python -m pip install pip-tools -U
Collecting pip-tools
  Downloading pip_tools-5.4.0-py2.py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 1.3 MB/s 
Requirement already satisfied, skipping upgrade: six in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (1.15.0)
Requirement already satisfied, skipping upgrade: click>=7 in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (7.1.2)
Requirement already satisfied, skipping upgrade: pip>=20.1 in ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from pip-tools) (20.2.3)
Installing collected packages: pip-tools
  Attempting uninstall: pip-tools
    Found existing installation: pip-tools 5.3.1
    Uninstalling pip-tools-5.3.1:
      Successfully uninstalled pip-tools-5.3.1
Successfully installed pip-tools-5.4.0
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.

$ echo myst-parser > myst.in

$ python -m piptools compile --generate-hashes --output-file myst.txt myst.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes --output-file=myst.txt myst.in
#
alabaster==0.7.12 \
    --hash=sha256:446438bdcca0e05bd45ea2de1668c1d9b032e1a9154c2c259092d77031ddd359 \
    --hash=sha256:a661d72d58e6ea8a57f7a86e37d86716863ee5e92788398526d58b26a4e4dc02 \
    # via sphinx
attrs==20.3.0 \
    --hash=sha256:31b2eced602aa8423c2aea9c76a724617ed67cf9513173fd3a4f03e3a929c7e6 \
    --hash=sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700 \
    # via markdown-it-py
babel==2.9.0 \
    --hash=sha256:9d35c22fcc79893c3ecc85ac4a56cde1ecf3f19c540bba0922308a6c06ca6fa5 \
    --hash=sha256:da031ab54472314f210b0adcff1588ee5d1d1d0ba4dbd07b94dba82bde791e05 \
    # via sphinx
certifi==2020.12.5 \
    --hash=sha256:1a4995114262bffbc2413b159f2a1a480c969de6e6eb13ee966d470af86af59c \
    --hash=sha256:719a74fb9e33b9bd44cc7f3a8d94bc35e4049deebe19ba7d8e108280cfd59830 \
    # via requests
chardet==3.0.4 \
    --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
    --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \
    # via requests
docutils==0.16 \
    --hash=sha256:0c5b78adfbf7762415433f5515cd5c9e762339e23369dbe8000d84a4bf4ab3af \
    --hash=sha256:c2de3a60e9e7d07be26b7f2b00ca0309c207e06c100f9cc2a94931fc75a478fc \
    # via myst-parser, sphinx
idna==2.10 \
    --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6 \
    --hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 \
    # via requests
imagesize==1.2.0 \
    --hash=sha256:6965f19a6a2039c7d48bca7dba2473069ff854c36ae6f19d2cde309d998228a1 \
    --hash=sha256:b1f6b5a4eab1f73479a50fb79fcf729514a900c341d8503d62a62dbc4127a2b1 \
    # via sphinx
jinja2==2.11.2 \
    --hash=sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0 \
    --hash=sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035 \
    # via sphinx
markdown-it-py==0.5.6 \
    --hash=sha256:6143d11221495edbf71beb7e455821ae6c8f0156710a1b11812662ed6dbd165b \
    --hash=sha256:dcfe4a0c6bef711cb6c42494ebf23a3cfe6f249bf995556498497dd8193bfc22 \
    # via myst-parser
markupsafe==1.1.1 \
    --hash=sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473 \
    --hash=sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161 \
    --hash=sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235 \
    --hash=sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5 \
    --hash=sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42 \
    --hash=sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff \
    --hash=sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b \
    --hash=sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1 \
    --hash=sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e \
    --hash=sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183 \
    --hash=sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66 \
    --hash=sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b \
    --hash=sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1 \
    --hash=sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15 \
    --hash=sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1 \
    --hash=sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e \
    --hash=sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b \
    --hash=sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905 \
    --hash=sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735 \
    --hash=sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d \
    --hash=sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e \
    --hash=sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d \
    --hash=sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c \
    --hash=sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21 \
    --hash=sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2 \
    --hash=sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5 \
    --hash=sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b \
    --hash=sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6 \
    --hash=sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f \
    --hash=sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f \
    --hash=sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2 \
    --hash=sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7 \
    --hash=sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be \
    # via jinja2
myst-parser==0.12.10 \
    --hash=sha256:4612c46196e0344bb7e49dbc3deb288f9b9a88fcf6e9f210f7f3ea5bc9899bfc \
    --hash=sha256:a5311da4398869e596250d5a93b523735c3beb8bc9d3eba853223c705802043b \
    # via -r myst.in
packaging==20.7 \
    --hash=sha256:05af3bb85d320377db281cf254ab050e1a7ebcbf5410685a9a407e18a1f81236 \
    --hash=sha256:eb41423378682dadb7166144a4926e443093863024de508ca5c9737d6bc08376 \
    # via sphinx
pygments==2.7.3 \
    --hash=sha256:ccf3acacf3782cbed4a989426012f1c535c9a90d3a7fc3f16d231b9372d2b716 \
    --hash=sha256:f275b6c0909e5dafd2d6269a656aa90fa58ebf4a74f8fcf9053195d226b24a08 \
    # via sphinx
pyparsing==2.4.7 \
    --hash=sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1 \
    --hash=sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b \
    # via packaging
pytz==2020.4 \
    --hash=sha256:3e6b7dd2d1e0a59084bcee14a17af60c5c562cdc16d828e8eba2e683d3a7e268 \
    --hash=sha256:5c55e189b682d420be27c6995ba6edce0c0a77dd67bfbe2ae6607134d5851ffd \
    # via babel
pyyaml==5.3.1 \
    --hash=sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97 \
    --hash=sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76 \
    --hash=sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2 \
    --hash=sha256:6034f55dab5fea9e53f436aa68fa3ace2634918e8b5994d82f3621c04ff5ed2e \
    --hash=sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648 \
    --hash=sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf \
    --hash=sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f \
    --hash=sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2 \
    --hash=sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee \
    --hash=sha256:ad9c67312c84def58f3c04504727ca879cb0013b2517c85a9a253f0cb6380c0a \
    --hash=sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d \
    --hash=sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c \
    --hash=sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a \
    # via myst-parser
requests==2.25.0 \
    --hash=sha256:7f1a0b932f4a60a1a65caa4263921bb7d9ee911957e0ae4a23a6dd08185ad5f8 \
    --hash=sha256:e786fa28d8c9154e6a4de5d46a1d921b8749f8b74e28bde23768e5e16eece998 \
    # via sphinx
snowballstemmer==2.0.0 \
    --hash=sha256:209f257d7533fdb3cb73bdbd24f436239ca3b2fa67d56f6ff88e86be08cc5ef0 \
    --hash=sha256:df3bac3df4c2c01363f3dd2cfa78cce2840a79b9f1c2d2de9ce8d31683992f52 \
    # via sphinx
sphinx==3.3.1 \
    --hash=sha256:1e8d592225447104d1172be415bc2972bd1357e3e12fdc76edf2261105db4300 \
    --hash=sha256:d4e59ad4ea55efbb3c05cde3bfc83bfc14f0c95aa95c3d75346fcce186a47960 \
    # via myst-parser
sphinxcontrib-applehelp==1.0.2 \
    --hash=sha256:806111e5e962be97c29ec4c1e7fe277bfd19e9652fb1a4392105b43e01af885a \
    --hash=sha256:a072735ec80e7675e3f432fcae8610ecf509c5f1869d17e2eecff44389cdbc58 \
    # via sphinx
sphinxcontrib-devhelp==1.0.2 \
    --hash=sha256:8165223f9a335cc1af7ffe1ed31d2871f325254c0423bc0c4c7cd1c1e4734a2e \
    --hash=sha256:ff7f1afa7b9642e7060379360a67e9c41e8f3121f2ce9164266f61b9f4b338e4 \
    # via sphinx
sphinxcontrib-htmlhelp==1.0.3 \
    --hash=sha256:3c0bc24a2c41e340ac37c85ced6dafc879ab485c095b1d65d2461ac2f7cca86f \
    --hash=sha256:e8f5bb7e31b2dbb25b9cc435c8ab7a79787ebf7f906155729338f3156d93659b \
    # via sphinx
sphinxcontrib-jsmath==1.0.1 \
    --hash=sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178 \
    --hash=sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8 \
    # via sphinx
sphinxcontrib-qthelp==1.0.3 \
    --hash=sha256:4c33767ee058b70dba89a6fc5c1892c0d57a54be67ddd3e7875a18d14cba5a72 \
    --hash=sha256:bd9fc24bcb748a8d51fd4ecaade681350aa63009a347a8c14e637895444dfab6 \
    # via sphinx
sphinxcontrib-serializinghtml==1.1.4 \
    --hash=sha256:eaa0eccc86e982a9b939b2b82d12cc5d013385ba5eadcc7e4fed23f4405f77bc \
    --hash=sha256:f242a81d423f59617a8e5cf16f5d4d74e28ee9a66f9e5b637a18082991db5a9a \
    # via sphinx
urllib3==1.26.2 \
    --hash=sha256:19188f96923873c92ccb987120ec4acaa12f0461fa9ce5d3d0772bc965a39e08 \
    --hash=sha256:d8ff90d979214d7b4f8ce956e80f4028fc6860e4431f731ea4a8c08f23f99473 \
    # via requests

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools
The generated requirements file may be rejected by pip install. See # WARNING lines for details.

$ python -m pip -V                                                           
pip 20.2.3 from ~/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pip (python 3.9)

$ python -m pip --use-feature=2020-resolver install -r myst.in -c myst.txt 
Collecting myst-parser
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    myst-parser from https://files.pythonhosted.org/packages/2f/f0/bf66b8c5428dd7352cbc258981dce95fcd3ae63e66aa843f3b0f5986c3ba/myst_parser-0.12.10-py3-none-any.whl#sha256=a5311da4398869e596250d5a93b523735c3beb8bc9d3eba853223c705802043b (from -r myst.in (line 1))
WARNING: You are using pip version 20.2.3; however, version 20.3.1 is available.
You should consider upgrading via the '~/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.

Additional information

Example crash: https://github.com/ansible/pylibssh/pull/157/checks?check_run_id=1511500384#step:9:15.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 8
  • Comments: 15 (14 by maintainers)

Commits related to this issue

Most upvoted comments

I’m on board for fixing the logic in the hash checking, to check that exactly 1 candidate is available, instead of checking that the requirement it got is a pin.

Beyond that, I don’t have any thoughts.

I’d be happy to help if someone wants to work on a pull request for this.

I’ve quickly looked into this, and found that _iter_candidates_from_constraints is likely/maybe meant to help with that, but does not yield a candidate, since the constraint has no links.

Code ref: https://github.com/pypa/pip/blob/9d25d2aba53928cb77149b8854b6b8a8ccfd81b8/src/pip/_internal/resolution/resolvelib/factory.py#L324-L345

Additional info / clarifying my use case

My use case is having a requirements.txt file with hashes (produced by pdm, pip-tools/pip-compile etc), and then using this for constraints to install some dependency out of it: req.txt:

pdm==1.9.0 \
    --hash=sha256:caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb \
    --hash=sha256:4de2f27bdce3a155fd56fb02f1c67d326ee24aa8083d8acaa2868e82d9c01999
  1. pip install -c req.txt pdm then fails with ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. (apparently having hashes in a constraints file enables require-hashes mode).

  2. With pip install -c req.txt pdm==1.9.0 (adding/copying the version from the constraint): ERROR: Hashes are required in --require-hashes mode, but they are missing from some requirements. … pdm==1.9.0 --hash=sha256:caa63e883561e5ab1815397eb7deae3837096bcea0349a19d04f72c8127a3feb).

Previous discussion: #9020