poetry: Poetry fails in CI/CD with ERROR: In --require-hashes mode, all requirements must have their versions pinned

I feel good enough about closing this now. The issue is with the change to how pip handles the constraints flag. pip maintainers have confirmed it’s not a bug but a design choice based on the new pip resolver.

Workarounds:

• For Hypermodern python users pass this flag in your poetry export command --without-hashes,
• Stop passing in packages with pinned hashes using the pip --contstraints flag.
• If you are using pip directly then pin it to a version before 20.3 python -m pip install --upgrade pip==20.2.4
• If you are using something else like virtualenv that depends on pip, make sure you pin it. python -m pip install --upgrade virtualenv==20.0.26 Or use a env var VIRTUALENV_PIP=20.2.4

I feel good enough about closing this now. The issue is with the change to how pip handles the constraints flag. pip maintainers have confirmed it’s not a bug but a design choice based on the new pip resolver.

Workarounds:

• For Hypermodern python users pass this flag in your poetry export command --without-hashes,

Perhaps a better solution to disabling hashes would be to disable the pip resolver. It doesn’t make sense to me that pip’s resolver would struggle with this when poetry has already done the dependency resolution. That’s what is exported to the requirements file. To do this in CI, do something like the following:

pip install --no-deps -r requirements.txt

This can be useful in some cases where you are using a combination of internal packages and PyPI packages and want to maintain the hashes for security reasons.

A better link for the pip-side of things – we consider this a bug and that it needs someone to pick it up and file a PR for it: https://github.com/pypa/pip/issues/9243

@shadycuz Be careful with pinning too eagerly. I guess you all know that by now. 😃 Additionally I do not believe pinning virtualenv has decisive influence on the pip version used.

I do not know if the message went through to you all, so I will repeat it here. From what I understood, the issue is triggered by having in the same pip install command requirements both with hashes and without hashes. So things like pip install Something --constraint export.txt will always fail if export.txt contains hashes, since Something has no hash on the command line. So indeed you can:

• either export without hashes
• or remove the Something and install everything with pip install --requirement export.txt

Both have their pros and cons. Altogether the integration between poetry and tox (nox) could be improved. I started a plug-in to attempt to fix this integration (tox-poetry-dev-dependencies), but it is only half way there.

Anyway, looks like for now the best compromise for your use cases might be to export without hashes.

Some things to look for in the future:

• Support dependencies groups (#1644) scheduled for v1.2
• https://github.com/python-poetry/poetry/issues/2010

https://github.com/pypa/pip/issues/9243 just got fixed by https://github.com/pypa/pip/pull/10962 thanks so much @q0w !

@shadycuz let me know what you find, because I gave up, would rather focus on my code right now

Still a problem on the poetry side. I’m not pinning anything, just trying to dockerise python, and the export command fails, despite none of https://github.com/python-poetry/poetry/issues/3472#issuecomment-744356551 being relevant for me: I’m not downgrading pip because of a poetry issue — this gotcha has to go instead. (And vice versa; shame on pip for breaking previous versions in a minor/patch upgrade!)

https://github.com/pypa/pip/issues/9243 was reopened so I think this ticket should be reopened again. I ran into this same issue while trying to automatize one pipeline today.

Wanted to quickly chime in that nox-poetry has just been updated to address this: https://github.com/cjolowicz/nox-poetry/releases/tag/v0.7.0

@sinoroc

1. Why is it filed as a poetry issue?

2. Read pypa/pip#8792 (and probably some other pip tickets as well)

I filed this as a poetry issue because I use poetry to manage my dependencies and not pip. I understand it’s most likely an issue with something other than poetry since I did not change the version of poetry when this problem occured, but since this issue affects other users of poetry it was probably a good place to create it in the end.

I’d prefer you walk me through the thinking process that lead you to using this pattern.

@trishankatdatadog and I both got the pattern from the excellent blog series Hypermodern Python. I’m not sure about @Corfucinas

• Why do you do what you do?
• Why do you export into requirements.txt then use this as a constraint file to install development dependencies?
• What are the other things you have tried and did not work?
• Why don’t you install all development dependencies?
• And so on

To be honest I do it because the blog told me so. I didn’t spend much time thinking about it. I know the blog has a companion repo. The companion repo seems to have evolved beyond what was instructed in the blog posts. It seems they found better ways to do things. I unfortunately didn’t realize this until after following the now outdated blog posts. I can say that up until this issue I was extremely happy with my new development workflow compared to just using pipand virtualenv in the past.

I have my answers for most of these questions, but they probably don’t match yours. Point is: maybe there is a better alternative.

If you have a fix I would be eager to test it, even if the fix is just changing how we install our development dependencies for nox.

This workflow still works fine locally. I’m guessing its something else that changed not related to poetry. I have confirmed that the github actions did change the underlying image used for it’s runners. I have issue with them as well.

@Corfucinas

I do have my hashes in requirements.txt

xdoctest==0.15.0 \
    --hash=sha256:695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e \
    --hash=sha256:7f0a184d403b69b166ebec1aadb13c98c96c59101e974ae2e4db4c3a803ec371

Thanks for posting this. I was running my builds in CI/CD and did not have access to the contents of the exported file. It’s great that we can confirm the hashes are present.

I can reproduce this issue, but it’s only related to the poetry export requirements.txt file and the specific use in nox (which was worked around by disabling security)

This isn’t something that can be fixed in peotry, but it could be something that peotry can provide a workaround for eg a new command like poetry contrained-venv-install ./venv/ flake8 flake8-black flake8-bugbear that uses the poetry lock file to update the specified venv with a subset of deps

This happened to me because of environment markers, I was running pip on a version of Python that didn’t match the tool.poetry.dependencies.python set in pyproject.toml. Therefore some dependencies were not specified for that Python version in requirements.txt, and pip would fail (it would implicitly look for a version of that lib to satisfy dependencies of other requirements, then fail with this obtuse message about hashes).

E.g. I have python = "^3.7" and my requirements.txt contains:

advocate==1.0.0
urllib3==1.26.4; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4" and python_version >= "3.7"

When running that through pip on Python 3.6, no version of urllib3 is pinned, but advocate depends on urllib3. Pip would grab some version of urllib3 that satisfies advocate’s dependencies and then complain that urllib3 does not specify hashes (the requirement it grabbed automatically has no hashes, even though the requirements.txt hash hashes).

Collecting urllib3<2.0,>=1.22
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    urllib3<2.0,>=1.22 from https://files.pythonhosted.org/packages/5f/64/43575537846896abac0b15c3e5ac678d787a4021e906703f1766bfb8ea11/urllib3-1.26.6-py2.py3-none-any.whl#sha256=39fb8672126159acb139a7718dd10806104dec1e2f0f6c88aab05d17df10c8d4 (from advocate==1.0.0->-r requirements.txt (line 1))

Hopefully that helps somebody else.

@Corfucinas @trishankatdatadog

Seems like this was already discussed and solved in the Hypermodern repo.

I’d prefer you walk me through the thinking process that lead you to using this pattern.

All good questions. It very largely came from hypermodern Python. I’ll dig deeper and see what broke.

What’s strange is that the nox setup works locally on my machine but not remotely on GitHub.

As for hashes in requirements.txt, I can confirm that when I rerun the offending command on my machine, I see the hashes.

I’m having the same problem.

This is my log

nox > Command pip install --constraint=/tmp/tmp2khsb66t xdoctest failed with exit code 1:
Collecting xdoctest
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    xdoctest from https://files.pythonhosted.org/packages/36/59/e70a32f7d5e2c4933bda64991ce5254f553c31ddd918f7b9a7e732cdd1e2/xdoctest-0.15.0-py2.py3-none-any.whl#sha256=695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e
nox > Session xdoctest-3.9.1 failed.

I do have my hashes in requirements.txt

xdoctest==0.15.0 \
    --hash=sha256:695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e \
    --hash=sha256:7f0a184d403b69b166ebec1aadb13c98c96c59101e974ae2e4db4c3a803ec371

@trishankatdatadog I had more time to investigate and while I haven’t found any fixes I do think I have started narrowing down the problem space. I still can’t rule out poetry for sure but it seems like it’s a pip or python issue from a change to the underlying Github actions servers. You can see a link to my issue with them above ^.