poetry: Poetry export does not include setuptools even when it is a direct dependency.
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvv
option).
- OS version and name: Fedora 31
- Poetry version: 1.0.0b4 (at the time of writing b5 is not on PyPi, will retry later on)
- Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/apollo13/1d0c72978531d3a5688f1e3847fa6fe6
Issue
Using the pyproject.toml
file from the Gist and executing poetry lock
I get the following lockfile:
[[package]]
category = "main"
description = "WSGI HTTP Server for UNIX"
name = "gunicorn"
optional = false
python-versions = ">=3.4"
version = "20.0.0"
[package.dependencies]
setuptools = ">=3.0"
[package.extras]
eventlet = ["eventlet (>=0.9.7)"]
gevent = ["gevent (>=0.13)"]
setproctitle = ["setproctitle"]
tornado = ["tornado (>=0.2)"]
[metadata]
content-hash = "20c8a1cc7d149b3ad58ca5063dfd37654f03ba65ee8b484d2faefe7ce28b7daf"
python-versions = "^3.7"
[metadata.files]
gunicorn = [
{file = "gunicorn-20.0.0-py2.py3-none-any.whl", hash = "sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb"},
{file = "gunicorn-20.0.0.tar.gz", hash = "sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50"},
]
setuptools is correctly discovered as dependency of gunicorn (see https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/setup.py#L71-L77 – it does indeed require it), but the lockfile does not contain any hashes for setuptools. If I now export to requirements.txt the file looks like this:
gunicorn==20.0.0 \
--hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
--hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50
Which is not installable with pip install --require-hashes --force-reinstall -r requirements.txt
(note the --require-hases
there):
Collecting gunicorn==20.0.0 (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/60/0d/3dbda0324f5bf007f3274e5ea09f0f3bcbf0ca01a75b80ff4f1ff9f8ecfd/gunicorn-20.0.0-py2.py3-none-any.whl
Collecting setuptools>=3.0 (from gunicorn==20.0.0->-r requirements.txt (line 1))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
setuptools>=3.0 from https://files.pythonhosted.org/packages/d9/de/554b6310ac87c5b921bc45634b07b11394fe63bc4cb5176f5240addf18ab/setuptools-41.6.0-py2.py3-none-any.whl#sha256=3e8e8505e563631e7cb110d9ad82d135ee866b8146d5efe06e42be07a72db20a (from gunicorn==20.0.0->-r requirements.txt (line 1))
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
If I instead add Django via poetry add Django
, the poetry.lock
file will contain Django’s dependencies (pytz
and sqlparse
):
pytz = [
{file = "pytz-2019.3-py2.py3-none-any.whl", hash = "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d"},
{file = "pytz-2019.3.tar.gz", hash = "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"},
]
sqlparse = [
{file = "sqlparse-0.3.0-py2.py3-none-any.whl", hash = "sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177"},
{file = "sqlparse-0.3.0.tar.gz", hash = "sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873"},
]
and also export them to the requirements file:
django==2.2.7 \
--hash=sha256:89c2007ca4fa5b351a51a279eccff298520783b713bf28efb89dfb81c80ea49b \
--hash=sha256:16040e1288c6c9f68c6da2fe75ebde83c0a158f6f5d54f4c5177b0c1478c5b86
gunicorn==20.0.0 \
--hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
--hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50
pytz==2019.3 \
--hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
--hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be
sqlparse==0.3.0 \
--hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
--hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873
I fear there is some special casing going on for setuptools
, it would be great if setuptools
also ended up in the lock and requirement files if it is a direct dependency of on of the added packages.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 32
- Comments: 32 (10 by maintainers)
Commits related to this issue
- Fix requirements.txt issue Why: The Python buildpack in Cloud.gov uses the requirements.txt file that we are exporting from Poetry. Unfortunately, the sha-256 hashes aren't working because of [some w... — committed to 18F/site-scanning by alexbielen 4 years ago
- Do not add dependencies to some ignored Poetry libraries Otherwise, the generated py_binary targets depend on non existing dependencies (such as setuptool). Poetry doesn't add several packages in th... — committed to nlewo/rules_poetry by nlewo 4 years ago
- Do not add dependencies to some ignored Poetry libraries Otherwise, the generated py_binary targets depend on non existing dependencies (such as setuptool). Poetry doesn't add several packages in th... — committed to nlewo/rules_poetry by nlewo 4 years ago
- Do not add dependencies to some ignored Poetry libraries Otherwise, the generated py_binary targets depend on non existing dependencies (such as setuptool). Poetry doesn't add several packages in th... — committed to nlewo/rules_poetry by nlewo 4 years ago
- Do not add dependencies to some ignored Poetry libraries (#2) Otherwise, the generated py_binary targets depend on non existing dependencies (such as setuptool). Poetry doesn't add several packag... — committed to soniaai/rules_poetry by nlewo 4 years ago
- Update requirements Generated with poetry export: $ poetry export --no-hashes -f requirements.txt > requirements.txt $ poetry export --no-hashes --dev -f requirements.txt > requirements-dev.... — committed to ilri/dspace-statistics-api by alanorth 4 years ago
- Update requirements Generated with poetry export: $ poetry export --no-hashes -f requirements.txt > requirements.txt $ poetry export --no-hashes --dev -f requirements.txt > requirements-dev.... — committed to ilri/dspace-statistics-api by alanorth 4 years ago
- Update requirements Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > require... — committed to ilri/dspace-statistics-api by alanorth 4 years ago
- Update requirements Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > require... — committed to ilri/dspace-statistics-api by alanorth 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Avoids version drift. The endpoint requirements.txt files did not contain the dependencies of each package, so `pip` would install the *latest*... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip` would install th... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requirements.txt files did not contain the recursive dependencies of each package, so `pip... — committed to linz/geostore by l0b0 4 years ago
- feat: Use poetry extras for subprojects (#74) * feat: Upgrade packages * feat: Use Poetry "extras" to keep dependencies separate Fixes issue #62. Avoids version drift. The endpoint requireme... — committed to linz/geostore by l0b0 4 years ago
- Update requirements Generated with poetry export: $ poetry export --without-hashes -f requirements.txt > requirements.txt $ poetry export --without-hashes --dev -f requirements.txt > require... — committed to ilri/dspace-statistics-api by alanorth 4 years ago
It seems there’s not too much reasoning behind this (https://github.com/pypa/pip/issues/6459)
At the very least, we should introduce an
--allow-unsafe
or similar flag, if not remove the unsafe packages hard-coding altogether.I am considering opening a PR for this. I plan to change the resolving behaviour so that the
UNSAFE_PACKAGES
are considered in the same way as any others but block them from being installed/exported unless an--allow-unsafe
flag is set.This will be my first poetry contribution so any advice, e.g. wrt anyone worth running this by before I start work, would be much appreciated.
This:
fixes my issue. Now the question is: Why are those unsafe and what are the consequences?
@bmw There exisits a merge request waiting final review that removes those unsafe packages completely. So there will be no need for such a flag.
Possibly poetry just should install and manage its setuptools and pip requirements in different/edited/hacked
sys.path
(…venv/lib/poetry-packages/ instead of site-packages), and then the venv itself can have no restrictions. IE: poetry should never have ‘unsafe’ packages, because its environment is separately managed by itself. That way we don’t have to wait until poetry no longer uses setuptools and/or pip. A side effect of this is that poetry can then bootstrap to use any support packages it needs (arrow or other fun things), because they won’t be a problem.@earonesty brings up an interesting point. I’d argue that
poetry
itself is usually installed globally or in another venv (ie viapipx
) so it should not conflict with anything in the venv at all. Does that address some of your concerns @sdispater? I am asking because this issue is the final show stopper for us to embrace poetry fully. I understand the motivation behind your argument, but ifpoetry
isn’t in the managed venv anyways, there are far less chances of issues. On the other hand, everyone installinggunicorn
ortensorflow
currently has problems withpoetry
.Same here. I found
VersionConflict
with thepip
andsetuptools
versions when trying to installtensorflow
andtensorboard
from poetry. The only workaround I found was to substitute thepoetry install
by:poetry run pip install -U pip
poetry run pip install -U setuptools
poetry install
This is not optimal as the pip and setuptools versions are not locked. Any solution to this? Seems to be a critical bug…
Issue #1651 seems to be tackling this point, although no activity since February
The reason why they are marked as “unsafe” is because Poetry relies on them internally (well mostly
pip
).However, Poetry will eventually no longer use them and when that happens this restriction will no longer be needed. Until then, I prefer that we leave it as it is to avoid unexpected side effects.
Hi, are there any updates on this? Just ran into the same issue when I specify https://github.com/pyeve/cerberus/blob/master/setup.py as a dependency
edit: I see it’s being fixed in an upcoming release https://github.com/python-poetry/poetry/issues/5351#issuecomment-1077304977
So I did dig through this issue again and came to the following conclusions:
distribute
is a very very old package (2013) that should not cause any problemssetuptools
andwheel
are required to install a source package inside a virtualenv. They are not required when wheels are installed (they are just extracted) or the source package contains a pyproject.toml (this explicitly trigger PEP-517 behavior, see https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support and causes build-isolation to be used, ie pip installs setuptools manually)pip
is needed inside the venv to actually install things; I’d strongly recommend to keep it this way because then pip easily knows what to install where. Please note that pip vendors basically everything (this includes setuptools, see https://github.com/pypa/pip/blob/master/src/pip/_vendor/vendor.txt for the full list) that is required for pip to function.So assuming we want to keep
pip
in the list of unsafe packages, I think we can freely removedistribute
and that leaves us withsetuptools
&wheel
. As noticed above, installing wheels requires neither of those two packages and the same applies to any source package with apyproject.toml
since that will build it’s own venv (more or less).So what to do about packages that use the “old” legacy build-format. Those do indeed require
setuptools
(and in the worst case a certain version), see the following example about installing Django:As you can see the virtualenv needs setuptools so pip can install Django, which does make sense since it needs to execute the
setup()
function insetup.py
. But as I said before the presence ofpyproject.toml
alone (even empty) would trigger a different behavior in pip (namely build isolation). Luckily this behavior can get forced one the cmdline:Eh voila, pip downloaded Django (it also downloaded
setuptools
&wheel
which you can see with-v
), created a.whl
file and installed that inside.The commandline option
--use-pep517
exists since pip 19.0 and is implicitly enabled for any project that contains apyproject.toml
(more and more). Using this optionpip
no longer has any dependency onsetuptools/wheel
in the venv. Unless poetry uses them somehow else in interesting ways I think we could remove them from the unsafe list.All in all this leads us to the following requirement: If poetry can require at least pip 19.0 and perform installations unconditionally with
pip install --use-pep517
then there does not seem to be any import ofsetuptools
orwheel
in the venv (I verified that by just removing those packages from the venv).I will try to adjust my local poetry with those changes and report back.
@bmw @apollo13 jfyi https://github.com/python-poetry/poetry-core/pull/168#issuecomment-817216870 if you add currently unsafe packages to your dependencies, older versions of poetry might not like it all that much.
Would
poetry
maintainers accept a PR adding a flag like--allow-unsafe
that would allow these “unsafe” packages to be included inpoetry.lock
?My particular use case is to generate a requirements file that I can use without
poetry
and I wantpip
,setuptools
, andwheel
pinned in the file.I’d be happy to try to open a PR if people like the idea of putting poetry deps in another folder (. poetry_deps ?)
@insysion Did you open a PR?
This is a serious bug. It causes the installation of older, buggier versions of setuptools like in https://github.com/pypa/setuptools/issues/1963. There’s no reason to break such projects.
Can confirm that setuptools as a dependency does not appear to be handled properly. There’s a package that depends on setuptools in my lockfile, but there is not associated entry in the lockfile for setuptools itself.
Lockfile snippet: