poetry: Poetry fails in CI/CD with ERROR: In --require-hashes mode, all requirements must have their versions pinned
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvv
option).
- OS version and name: Ubuntu 18.04.5 LTS
- Poetry version: 1.1.4
- Link of a Gist with the contents of your pyproject.toml file: https://github.com/DontShaveTheYak/cloud-radar/pull/5
Issue
My builds were working fine locally and remotely. Here is an example of the first 12 or so lines…
2020-11-28T13:25:00.7733264Z ##[group]Run nox
2020-11-28T13:25:00.7733666Z [36;1mnox[0m
2020-11-28T13:25:00.7772013Z shell: /bin/bash -e {0}
2020-11-28T13:25:00.7772325Z env:
2020-11-28T13:25:00.7772772Z pythonLocation: /opt/hostedtoolcache/Python/3.8.6/x64
2020-11-28T13:25:00.7773219Z ##[endgroup]
2020-11-28T13:25:00.8815564Z nox > Running session lint-3.9
2020-11-28T13:25:00.8867576Z nox > Session lint-3.9 skipped: Python interpreter 3.9 not found.
2020-11-28T13:25:00.8869448Z nox > Running session lint-3.8
2020-11-28T13:25:00.8870400Z nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint-3-8
2020-11-28T13:25:01.6319387Z nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp2554a7nu
2020-11-28T13:25:04.4828117Z nox > pip install --constraint=/tmp/tmp2554a7nu flake8 flake8-black flake8-bugbear flake8-import-order
2020-11-28T13:25:11.0176722Z nox > flake8 src tests noxfile.py
I added a package with poetry add cfn-flip
My builds still work locally but fail in github actions:
2020-12-10T12:32:26.6270480Z ##[group]Run nox
2020-12-10T12:32:26.6270955Z [36;1mnox[0m
2020-12-10T12:32:26.6313567Z shell: /bin/bash -e {0}
2020-12-10T12:32:26.6313932Z env:
2020-12-10T12:32:26.6314480Z pythonLocation: /opt/hostedtoolcache/Python/3.8.6/x64
2020-12-10T12:32:26.6315036Z ##[endgroup]
2020-12-10T12:32:26.7301178Z nox > Running session lint-3.9
2020-12-10T12:32:26.7352891Z nox > Session lint-3.9 skipped: Python interpreter 3.9 not found.
2020-12-10T12:32:26.7354096Z nox > Running session lint-3.8
2020-12-10T12:32:26.7357585Z nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint-3-8
2020-12-10T12:32:27.5320470Z nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp2k2e6e06
2020-12-10T12:32:30.7810961Z nox > pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order
2020-12-10T12:32:32.4291735Z nox > Command pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order failed with exit code 1:
2020-12-10T12:32:32.4293643Z Collecting flake8
2020-12-10T12:32:32.4300409Z ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
2020-12-10T12:32:32.4303405Z flake8 from https://files.pythonhosted.org/packages/d4/ca/3971802ee6251da1abead1a22831d7f4743781e2f743bd266bdd2f46c19b/flake8-3.8.4-py2.py3-none-any.whl#sha256=749dbbd6bfd0cf1318af27bf97a14e28e5ff548ef8e5b1566ccfb25a11e7c839
Since my builds still worked locally I thought it might be an issue with PyPi, so I waited a day, but they are still failing. Up above I have linked my PR so you should be able to see the changes to the toml and the lock file. I’m very unsure what to try next except maybe deleting the lock file and having poetry recreate it again.
Locally and in the CI/CD I’m running the same Nox and Poetry versions. I think this might be related to python-poetry/poetry-plugin-export#38 and possibly python-poetry/poetry-plugin-export#145 since both of those are about poetry export
.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 4
- Comments: 32 (8 by maintainers)
Commits related to this issue
- Remove hashes from requirements to fix pip install See https://github.com/python-poetry/poetry/issues/3472 — committed to sergeyklay/branch by sergeyklay 3 years ago
- Export poetry dependenciese to requirements.txt without hashes * hashes may be problematic with poetry - https://github.com/python-poetry/poetry/issues/3472 - https://github.com/python-poetry... — committed to appleparan/mise.py by appleparan 3 years ago
- Fix requirements has/pinning issue SEE: https://github.com/python-poetry/poetry/issues/3472 — committed to Apreche/frontrowcrew by Apreche 3 years ago
- All core models, views, and placeholder templates created and tested. (#53) * All core models, views, and placeholder templates created and tested. * Fix linting to ignore generated file(s) and Dj... — committed to Apreche/frontrowcrew by Apreche 3 years ago
- Fix compose and demon dockerfiles Workaround --without-hashes for https://github.com/python-poetry/poetry/issues/3472 — committed to intermine/intermine_cloud by heralden 2 years ago
- ci: fix test_image step https://github.com/python-poetry/poetry/issues/3472 — committed to mriedmann/pipecheck by mriedmann 2 years ago
- fix: add --without-hashes to fix https://github.com/python-poetry/poetry/issues/3472 — committed to Greenroom-Robotics/platform_cli by MrBlenny 2 years ago
- chore(release): 1.0.1 [skip ci] ## [1.0.1](https://github.com/Greenroom-Robotics/platform_cli/compare/v1.0.0...v1.0.1) (2022-11-20) ### Bug Fixes * add __init__.py to git asset list ([fde2453](http... — committed to Greenroom-Robotics/platform_cli by semantic-release-bot 2 years ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
- fix: Update noxfile and dependencies Update dependencies, update minimal python version to 3.6.1 Fix noxfile, generate constraint file using constraint format add `--without-hashes` option to be able... — committed to boatx/SockIt by boatx 10 months ago
I feel good enough about closing this now. The issue is with the change to how
pip
handles the constraints flag.pip
maintainers have confirmed it’s not a bug but a design choice based on the newpip
resolver.Workarounds:
poetry export
command--without-hashes,
pip --contstraints
flag.python -m pip install --upgrade pip==20.2.4
virtualenv
that depends on pip, make sure you pin it.python -m pip install --upgrade virtualenv==20.0.26
Or use a env varVIRTUALENV_PIP=20.2.4
Perhaps a better solution to disabling hashes would be to disable the pip resolver. It doesn’t make sense to me that pip’s resolver would struggle with this when poetry has already done the dependency resolution. That’s what is exported to the requirements file. To do this in CI, do something like the following:
This can be useful in some cases where you are using a combination of internal packages and PyPI packages and want to maintain the hashes for security reasons.
A better link for the pip-side of things – we consider this a bug and that it needs someone to pick it up and file a PR for it: https://github.com/pypa/pip/issues/9243
@shadycuz Be careful with pinning too eagerly. I guess you all know that by now. 😃 Additionally I do not believe pinning virtualenv has decisive influence on the pip version used.
I do not know if the message went through to you all, so I will repeat it here. From what I understood, the issue is triggered by having in the same
pip install
command requirements both with hashes and without hashes. So things likepip install Something --constraint export.txt
will always fail ifexport.txt
contains hashes, sinceSomething
has no hash on the command line. So indeed you can:Something
and install everything withpip install --requirement export.txt
Both have their pros and cons. Altogether the integration between poetry and tox (nox) could be improved. I started a plug-in to attempt to fix this integration (tox-poetry-dev-dependencies), but it is only half way there.
Anyway, looks like for now the best compromise for your use cases might be to export without hashes.
Some things to look for in the future:
https://github.com/pypa/pip/issues/9243 just got fixed by https://github.com/pypa/pip/pull/10962 thanks so much @q0w !
@shadycuz let me know what you find, because I gave up, would rather focus on my code right now
Still a problem on the poetry side. I’m not pinning anything, just trying to dockerise python, and the export command fails, despite none of https://github.com/python-poetry/poetry/issues/3472#issuecomment-744356551 being relevant for me: I’m not downgrading pip because of a poetry issue — this gotcha has to go instead. (And vice versa; shame on pip for breaking previous versions in a minor/patch upgrade!)
https://github.com/pypa/pip/issues/9243 was reopened so I think this ticket should be reopened again. I ran into this same issue while trying to automatize one pipeline today.
Wanted to quickly chime in that
nox-poetry
has just been updated to address this: https://github.com/cjolowicz/nox-poetry/releases/tag/v0.7.0@sinoroc
I filed this as a
poetry
issue because I usepoetry
to manage my dependencies and notpip
. I understand it’s most likely an issue with something other thanpoetry
since I did not change the version ofpoetry
when this problem occured, but since this issue affects other users ofpoetry
it was probably a good place to create it in the end.@trishankatdatadog and I both got the pattern from the excellent blog series Hypermodern Python. I’m not sure about @Corfucinas
To be honest I do it because the blog told me so. I didn’t spend much time thinking about it. I know the blog has a companion repo. The companion repo seems to have evolved beyond what was instructed in the blog posts. It seems they found better ways to do things. I unfortunately didn’t realize this until after following the now outdated blog posts. I can say that up until this issue I was extremely happy with my new development workflow compared to just using
pip
andvirtualenv
in the past.If you have a fix I would be eager to test it, even if the fix is just changing how we install our development dependencies for
nox
.This workflow still works fine locally. I’m guessing its something else that changed not related to
poetry
. I have confirmed that the github actions did change the underlying image used for it’s runners. I have issue with them as well.@Corfucinas
Thanks for posting this. I was running my builds in CI/CD and did not have access to the contents of the exported file. It’s great that we can confirm the hashes are present.
I can reproduce this issue, but it’s only related to the poetry export requirements.txt file and the specific use in nox (which was worked around by disabling security)
This isn’t something that can be fixed in peotry, but it could be something that peotry can provide a workaround for eg a new command like
poetry contrained-venv-install ./venv/ flake8 flake8-black flake8-bugbear
that uses the poetry lock file to update the specified venv with a subset of depsThis happened to me because of environment markers, I was running pip on a version of Python that didn’t match the
tool.poetry.dependencies.python
set inpyproject.toml
. Therefore some dependencies were not specified for that Python version inrequirements.txt
, and pip would fail (it would implicitly look for a version of that lib to satisfy dependencies of other requirements, then fail with this obtuse message about hashes).E.g. I have
python = "^3.7"
and my requirements.txt contains:When running that through pip on Python 3.6, no version of
urllib3
is pinned, butadvocate
depends onurllib3
. Pip would grab some version ofurllib3
that satisfiesadvocate
’s dependencies and then complain thaturllib3
does not specify hashes (the requirement it grabbed automatically has no hashes, even though the requirements.txt hash hashes).Hopefully that helps somebody else.
@Corfucinas @trishankatdatadog
Seems like this was already discussed and solved in the Hypermodern repo.
All good questions. It very largely came from hypermodern Python. I’ll dig deeper and see what broke.
What’s strange is that the nox setup works locally on my machine but not remotely on GitHub.
As for hashes in requirements.txt, I can confirm that when I rerun the offending command on my machine, I see the hashes.
I’m having the same problem.
This is my log
I do have my hashes in
requirements.txt
@trishankatdatadog I had more time to investigate and while I haven’t found any fixes I do think I have started narrowing down the problem space. I still can’t rule out poetry for sure but it seems like it’s a
pip
or python issue from a change to the underlying Github actions servers. You can see a link to my issue with them above ^.