poetry: Poetry export does not include setuptools even when it is a direct dependency.

It seems there’s not too much reasoning behind this (https://github.com/pypa/pip/issues/6459)

At the very least, we should introduce an --allow-unsafe or similar flag, if not remove the unsafe packages hard-coding altogether.

I am considering opening a PR for this. I plan to change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.

This will be my first poetry contribution so any advice, e.g. wrt anyone worth running this by before I start work, would be much appreciated.

This:

diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py
index a077e28..1b13212 100644
--- a/poetry/puzzle/provider.py
+++ b/poetry/puzzle/provider.py
@@ -60,7 +60,7 @@ class Indicator(ProgressIndicator):
 
 class Provider:
 
-    UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"}
+    UNSAFE_PACKAGES = {}
 
     def __init__(self, package, pool, io):  # type: (Package, Pool, Any) -> None
         self._package = package

fixes my issue. Now the question is: Why are those unsafe and what are the consequences?

@bmw There exisits a merge request waiting final review that removes those unsafe packages completely. So there will be no need for such a flag.

Possibly poetry just should install and manage its setuptools and pip requirements in different/edited/hacked sys.path (…venv/lib/poetry-packages/ instead of site-packages), and then the venv itself can have no restrictions. IE: poetry should never have ‘unsafe’ packages, because its environment is separately managed by itself. That way we don’t have to wait until poetry no longer uses setuptools and/or pip. A side effect of this is that poetry can then bootstrap to use any support packages it needs (arrow or other fun things), because they won’t be a problem.

@earonesty brings up an interesting point. I’d argue that poetry itself is usually installed globally or in another venv (ie via pipx) so it should not conflict with anything in the venv at all. Does that address some of your concerns @sdispater? I am asking because this issue is the final show stopper for us to embrace poetry fully. I understand the motivation behind your argument, but if poetry isn’t in the managed venv anyways, there are far less chances of issues. On the other hand, everyone installing gunicorn or tensorflow currently has problems with poetry.

Same here. I found VersionConflict with the pip and setuptools versions when trying to install tensorflow and tensorboard from poetry. The only workaround I found was to substitute the poetry install by:

poetry run pip install -U pip poetry run pip install -U setuptools poetry install

This is not optimal as the pip and setuptools versions are not locked. Any solution to this? Seems to be a critical bug…

Issue #1651 seems to be tackling this point, although no activity since February

The reason why they are marked as “unsafe” is because Poetry relies on them internally (well mostly pip).

However, Poetry will eventually no longer use them and when that happens this restriction will no longer be needed. Until then, I prefer that we leave it as it is to avoid unexpected side effects.

Hi, are there any updates on this? Just ran into the same issue when I specify https://github.com/pyeve/cerberus/blob/master/setup.py as a dependency

edit: I see it’s being fixed in an upcoming release https://github.com/python-poetry/poetry/issues/5351#issuecomment-1077304977

So I did dig through this issue again and came to the following conclusions:

• distribute is a very very old package (2013) that should not cause any problems
• setuptools and wheel are required to install a source package inside a virtualenv. They are not required when wheels are installed (they are just extracted) or the source package contains a pyproject.toml (this explicitly trigger PEP-517 behavior, see https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support and causes build-isolation to be used, ie pip installs setuptools manually)
• pip is needed inside the venv to actually install things; I’d strongly recommend to keep it this way because then pip easily knows what to install where. Please note that pip vendors basically everything (this includes setuptools, see https://github.com/pypa/pip/blob/master/src/pip/_vendor/vendor.txt for the full list) that is required for pip to function.

So assuming we want to keep pip in the list of unsafe packages, I think we can freely remove distribute and that leaves us with setuptools & wheel. As noticed above, installing wheels requires neither of those two packages and the same applies to any source package with a pyproject.toml since that will build it’s own venv (more or less).

So what to do about packages that use the “old” legacy build-format. Those do indeed require setuptools (and in the worst case a certain version), see the following example about installing Django:

➜  /tmp python -mvenv testing
➜  /tmp ./testing/bin/pip uninstall wheel setuptools
WARNING: Skipping wheel as it is not installed.
Found existing installation: setuptools 49.1.3
Uninstalling setuptools-49.1.3:
  Would remove:
    /tmp/testing/bin/easy_install
    /tmp/testing/bin/easy_install-3.9
    /tmp/testing/lib/python3.9/site-packages/easy_install.py
    /tmp/testing/lib/python3.9/site-packages/pkg_resources/*
    /tmp/testing/lib/python3.9/site-packages/setuptools-49.1.3.dist-info/*
    /tmp/testing/lib/python3.9/site-packages/setuptools/*
Proceed (y/n)? y
  Successfully uninstalled setuptools-49.1.3
➜  /tmp ./testing/bin/pip install --no-binary Django Django
Collecting Django
  Using cached Django-3.1.6.tar.gz (9.6 MB)
    ERROR: Command errored out with exit status 1:
     command: /tmp/testing/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gckw7rgk/django/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gckw7rgk/django/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-a61yhryq
         cwd: /tmp/pip-install-gckw7rgk/django/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

As you can see the virtualenv needs setuptools so pip can install Django, which does make sense since it needs to execute the setup() function in setup.py. But as I said before the presence of pyproject.toml alone (even empty) would trigger a different behavior in pip (namely build isolation). Luckily this behavior can get forced one the cmdline:

➜  /tmp ./testing/bin/pip install --use-pep517 --no-binary Django Django
Collecting Django
  Using cached Django-3.1.6.tar.gz (9.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
Collecting pytz
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting asgiref<4,>=3.2.10
  Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
Building wheels for collected packages: Django
  Building wheel for Django (PEP 517) ... done
  Created wheel for Django: filename=Django-3.1.6-py3-none-any.whl size=7834188 sha256=b2a4a23721a1ac19cdfb763d7cab620c5a8d16bf35f754259e33ccd461e45ba3
  Stored in directory: /home/florian/.cache/pip/wheels/0d/89/6b/c66c6f61ef318e1e11cbc16aead880ccacca0360e2810a440c
Successfully built Django
Installing collected packages: sqlparse, pytz, asgiref, Django
Successfully installed Django-3.1.6 asgiref-3.3.1 pytz-2021.1 sqlparse-0.4.1

Eh voila, pip downloaded Django (it also downloaded setuptools & wheel which you can see with -v), created a .whl file and installed that inside.

The commandline option --use-pep517 exists since pip 19.0 and is implicitly enabled for any project that contains a pyproject.toml (more and more). Using this option pip no longer has any dependency on setuptools/wheel in the venv. Unless poetry uses them somehow else in interesting ways I think we could remove them from the unsafe list.

All in all this leads us to the following requirement: If poetry can require at least pip 19.0 and perform installations unconditionally with pip install --use-pep517 then there does not seem to be any import of setuptools or wheel in the venv (I verified that by just removing those packages from the venv).

I will try to adjust my local poetry with those changes and report back.

@bmw @apollo13 jfyi https://github.com/python-poetry/poetry-core/pull/168#issuecomment-817216870 if you add currently unsafe packages to your dependencies, older versions of poetry might not like it all that much.

Would poetry maintainers accept a PR adding a flag like --allow-unsafe that would allow these “unsafe” packages to be included in poetry.lock?

My particular use case is to generate a requirements file that I can use without poetry and I want pip, setuptools, and wheel pinned in the file.

I’d be happy to try to open a PR if people like the idea of putting poetry deps in another folder (. poetry_deps ?)

@insysion Did you open a PR?

This is a serious bug. It causes the installation of older, buggier versions of setuptools like in https://github.com/pypa/setuptools/issues/1963. There’s no reason to break such projects.

Can confirm that setuptools as a dependency does not appear to be handled properly. There’s a package that depends on setuptools in my lockfile, but there is not associated entry in the lockfile for setuptools itself.

Lockfile snippet:

[[package]]
category = "main"
description = "Library to implement a well-behaved Unix daemon process."
name = "python-daemon"
optional = false
python-versions = "*"
version = "2.2.4"

[package.dependencies]
docutils = "*"
lockfile = ">=0.10"
setuptools = "*"