poetry: 1.0.10 -> 1.1 regression: EnvCommandError: 'No module named setuptools' on Python 3.7

In https://github.com/python-poetry/poetry/issues/1135#issuecomment-719902456 @NeilGirdhar pointed out that one would need to change pyproject.toml to include setuptools:

[build-system]
-requires = ["poetry_core>=1.0"]
+requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"

With this added in my above example (reproduce-3153.zip) it now builds:

poetry run pip install -e .
Obtaining file:///Users/kristofferb/Code/temp/gist-reproduce-3153
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: reproduce-issue
  Attempting uninstall: reproduce-issue
    Found existing installation: reproduce-issue 0.1.0
    Uninstalling reproduce-issue-0.1.0:
      Successfully uninstalled reproduce-issue-0.1.0
  Running setup.py develop for reproduce-issue
Successfully installed reproduce-issue

And indeed, as others have mentioned in https://github.com/python-poetry/poetry/issues/34, this is not a poetry issue. But it seems like many would like poetry to have a workaround.

same issue, wrapt depends on it, manually removing setuptools from the poetry.lock worked for me

diff --git a/poetry.lock b/poetry.lock
index c8f8357..f858bfe 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -868,18 +868,6 @@ botocore = ">=1.12.36,<2.0a.0"
 [package.extras]
 crt = ["botocore[crt] (>=1.20.29,<2.0a.0)"]

-[[package]]
-name = "setuptools"
-version = "57.4.0"
-description = "Easily download, build, install, upgrade, and uninstall Python packages"
-category = "main"
-optional = false
-python-versions = ">=3.6"
-
-[package.extras]
-docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "sphinx-inline-tabs", "sphinxcontrib-towncrier", "furo"]
-testing = ["pytest (>=4.6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "mock", "flake8-2020", "virtualenv (>=13.0.0)", "pytest-virtualenv (>=1.2.7)", "wheel", "paver", "pip (>=19.1)", "jaraco.envs", "pytest-xdist", "sphinx", "jaraco.path (>=3.2.0)", "pytest-black (>=0.3.7)", "pytest-mypy"]
-
 [[package]]
 name = "six"
 version = "1.16.0"
@@ -1555,10 +1543,6 @@ s3transfer = [
     {file = "s3transfer-0.5.0-py3-none-any.whl", hash = "sha256:9c1dc369814391a6bda20ebbf4b70a0f34630592c9aa520856bf384916af2803"},
     {file = "s3transfer-0.5.0.tar.gz", hash = "sha256:50ed823e1dc5868ad40c8dc92072f757aa0e653a192845c94a3b676f4a62da4c"},
 ]
-setuptools = [
-    {file = "setuptools-57.4.0-py3-none-any.whl", hash = "sha256:a49230977aa6cfb9d933614d2f7b79036e9945c4cdd7583163f4e920b83418d6"},
-    {file = "setuptools-57.4.0.tar.gz", hash = "sha256:6bac238ffdf24e8806c61440e755192470352850f3419a52f26ffe0a1a64f465"},
-]
 six = [
     {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"},
     {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"},

> rm -rf "$(poetry env info --path)"
> poetry install
...
Installing wrapt (1.12.1)

This still happens on a fresh install of Poetry (1.1.13) with Python 3.10.2 (on Arch Linux, at least). I tried to follow the poetry-demo tutorial but it exploded when I attempted:

poetry add pendulum

Adding setuptools to the [build-system] section doesn’t fix it. Note that I’m using the system-provided versions of pip (20.3.4) and setuptools (59.3.0), although the virtualenv seems to be pulling it’s own version of pip anyway (21.3.1).

Update: Actually the problem may be with pendulum itself. I was just able to do a poetry add more-itertools without issue.

For folks facing this issue, can you please try master? Note that you will need to use a new virtual environment or run poetry install --remove-untracked on existing environments.

Using pipx

pipx install --force --suffix=@master 'poetry @ git+https://github.com/python-poetry/poetry.git@master'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@master
poetry new foobar
pushd foobar
# commands to reproduce issue 
EOF

I can confirm this is a problem and affects anything that ultimately still depends on calling python setup.py, such as:

• installing a poetry powered package with pip install -e
• using a local package in pyproject.toml like @jonapich demonstrates with test-testing = { path = "../test-testing/" }

I can confirm the problem with Python 3.6, 3.7 and 3.8 alike.

And this is really hard to debug because even if you literally copy and paste the command it’s trying to run, it just works, in my case for a sample package:

[EnvCommandError]
Command ['/home/gustavo/.cache/pypoetry/virtualenvs/package-94uOVQQp-py3.8/bin/pip', 'install', '--no-deps', '-U', '-e', '/home/gustavo/my-package/package/../packb'] errored with the following return code 1, and output: 
Looking in indexes: https://pypi.org/simple, Obtaining file:///home/gustavo/my-package/packb
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Installing collected packages: packb
  Running setup.py develop for packb
    ERROR: Command errored out with exit status 1:
     command: /home/gustavo/.cache/pypoetry/virtualenvs/package-94uOVQQp-py3.8/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/gustavo/my-package/packb/setup.py'"'"'; __file__='"'"'/home/gustavo/my-package/packb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/gustavo/my-package/packb/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/gustavo/.cache/pypoetry/virtualenvs/package-94uOVQQp-py3.8/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/gustavo/my-package/packb/setup.py'"'"'; __file__='"'"'/home/gustavo/my-package/packb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

But if I simply copy and paste the command below, it runs ok:

○ → /home/gustavo/.cache/pypoetry/virtualenvs/package-94uOVQQp-py3.8/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/gustavo/my-package/packb/setup.py'"'"'; __file__='"'"'/home/gustavo/my-package/packb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
running develop
running egg_info
creating packb.egg-info
writing packb.egg-info/PKG-INFO
writing dependency_links to packb.egg-info/dependency_links.txt
writing top-level names to packb.egg-info/top_level.txt
writing manifest file 'packb.egg-info/SOURCES.txt'
reading manifest file 'packb.egg-info/SOURCES.txt'
writing manifest file 'packb.egg-info/SOURCES.txt'
running build_ext
Creating /home/gustavo/.cache/pypoetry/virtualenvs/package-94uOVQQp-py3.8/lib/python3.8/site-packages/packb.egg-link (link to .)
Adding packb 0.0.0 to easy-install.pth file

Installed /home/gustavo/my-package/package

To make this “work” I had to install bogus setup.py files to my packages, such as:

from setuptools import setup
setup(name="packb")

Yet it still fails when running from within poetry.

@abn good point. Apparently I have 1.2.0a installed for some reason. Thanks for pointing that out! Downgrading to 1.1.11 fixes the problem.

I’m seeing this in Github Actions now (example: https://github.com/jrnl-org/jrnl/runs/3016985092). Has anyone found a workaround?

Also, I noticed that the first line after running poetry install is:

    • Removing setuptools (57.0.0)

And this still happens even if I remove the --remove-untracked flag. Interestingly, though, this error isn’t happening when I install poetry v1.2.0a1. So maybe it’s already fixed somehow?

Some things I tried:

• Manually install setuptools (said it was already installed)
• Removed --remove-untracked flag (still see the “removing setuptools” line)
• Updated lock file and verified that setuptools is listed

Here are a bunch of CI runs that show the issue, in case it helps:

• https://github.com/jrnl-org/jrnl/actions/runs/3016835445
• https://github.com/jrnl-org/jrnl/actions/runs/1010876695
• https://github.com/jrnl-org/jrnl/actions/runs/1010902188
• https://github.com/jrnl-org/jrnl/actions/runs/1010906595
• https://github.com/jrnl-org/jrnl/actions/runs/1010927665

Hi, if I understood the status of this issue correctly, a workaround (for the obscure setuptools problem, which will not go away until poetry no longer uses setuptools) is on main, but not released yet, and for a package which is built by poetry, but which has a setup.py (for example, setup.py which is used to install the sdist), one can workaround until the next release, by adding

[build-system]
-requires = ["poetry_core>=1.0"]
+requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"

as suggested by @kbakk. Is this correct?

I don’t understand completely what’s going on here. But the root cause seems to be, that setuptools started to vendor distutils in a way, that whenever a module imports distutils it gets redirected to the vendored version of setuptools. And that seems not work in any case. See https://github.com/pypa/distutils/issues/17

So the problem is not, that setuptools is not installed. The problem is, that the version of setuptools that comes with the isolated build environment doesn’t work as expected. A workaround is to run export SETUPTOOLS_USE_DISTUTILS=stdlib before a pip install.

So at it stands now, it looks to me a setuptools problem and not a poetry problem.

I went through the trouble of removing everything python and poetry off my system. I then reinstalled py3.8 using the latest installer and then reinstalled poetry using the Invoke-Webrequest method from the docs.

The behavior is still the same. Some observations:

• Through poetry run python -m pip list I can see setuptools is installed in the project’s virtual environment.
• If I launch poetry run python I can import distutils.util

This is where poetry breaks it for me even though python/pip can handle it:

• If I launch poetry shell and then proceed to pip install ../test-testing/ it will fail.
• If I create my own environment python -m venv virtualenv and then call virtualenv/Scripts/pip install ../test-testing/ it works.

Additional note: The dependencies in test-testing have no effect on the issue. I tried to remove them all but I’m still unable to install it if poetry is involved with it.

I am also getting this error and I can reproduce it.

https://github.com/jonapich/test-poetry-setuptools

Clone that repo, and try “poetry install” in the test-functools folder, you will get the same stacktrace as in the bug report.

This seems pretty serious 🤔

I’m on 1.1.3, Windows, using powershell. I installed using the get-poetry bootstrap.

I have confirmed that this bug appears when going from 1.0.10 -> 1.1 and it was also working in 1.0.9

Also, I never used python 3.7; only 3.6 and 3.8.

I ran into this issue with Poetry 1.2.0a2 as it adds setuptools to the lock file when 1.1.x would not. Relocking the project with 1.1.8 removes the setuptools dependency and resolves my issue.

@jtzero yes after removing setuptools in poetry.lock its working now. Thank you.!!!

i am also getting same issue even with this setup

requires = ["setuptools","poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

I can’t reproduce this now. I’ll close it out unless someone else runs into it on 1.1.4.

@jonapich based on your description I have attempted to recreate the scenario (minus the windows environment). Let me know what needs to be changed to reproduce the issue.

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -ex
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - --version 1.0.10
source ~/.poetry/env
poetry --version
git clone https://github.com/jonapich/test-poetry-setuptools.git
pushd test-poetry-setuptools/test-functools/
poetry lock
poetry install
poetry self update
poetry install
rm -rf \$(poetry env info -p)
poetry --version
poetry install
rm -rf \$(poetry env info -p)
git checkout poetry.lock
poetry install
EOF

+ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
+ python - --version 1.0.10
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

$HOME/.poetry/bin

This path will then be added to your `PATH` environment variable by
modifying the profile file located at:

$HOME/.profile

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing version: 1.0.10
  - Downloading poetry-1.0.10-linux.tar.gz (29.66MB)

Poetry (1.0.10) is installed now. Great!

To get started you need Poetry's bin directory ($HOME/.poetry/bin) in your `PATH`
environment variable. Next time you log in this will be done
automatically.

To configure your current shell run `source $HOME/.poetry/env`

+ source /root/.poetry/env
++ export PATH=/root/.poetry/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/root/.poetry/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ poetry --version
Poetry version 1.0.10
+ git clone https://github.com/jonapich/test-poetry-setuptools.git
Cloning into 'test-poetry-setuptools'...
+ pushd test-poetry-setuptools/test-functools/
/test-poetry-setuptools/test-functools /
+ poetry lock
Creating virtualenv test-functools-HE92XDqH-py3.8 in /root/.cache/pypoetry/virtualenvs
The lock file might not be compatible with the current version of Poetry.
Upgrade Poetry to ensure the lock file is read properly or, alternatively, regenerate the lock file with the `poetry lock` command.
Updating dependencies
Resolving dependencies...

Writing lock file
+ poetry install
Installing dependencies from lock file


Package operations: 15 installs, 0 updates, 0 removals

  - Installing pyparsing (2.4.7)
  - Installing six (1.15.0)
  - Installing attrs (20.2.0)
  - Installing iniconfig (1.1.1)
  - Installing packaging (20.4)
  - Installing pluggy (0.13.1)
  - Installing py (1.9.0)
  - Installing toml (0.10.1)
  - Installing mypy-extensions (0.4.3)
  - Installing pytest (6.1.1)
  - Installing typed-ast (1.4.1)
  - Installing typing-extensions (3.7.4.3)
  - Installing inflection (0.5.1)
  - Installing mypy (0.790)
  - Installing test-testing (0.1.4 ../test-testing)
  - Installing test-functools (0.1.4)
+ poetry self update
Updating to 1.1.3
 - Downloading poetry-1.1.3-linux.tar.gz 0%
 - Downloading poetry-1.1.3-linux.tar.gz 10%
 - Downloading poetry-1.1.3-linux.tar.gz 20%
 - Downloading poetry-1.1.3-linux.tar.gz 30%
 - Downloading poetry-1.1.3-linux.tar.gz 40%
 - Downloading poetry-1.1.3-linux.tar.gz 50%
 - Downloading poetry-1.1.3-linux.tar.gz 60%
 - Downloading poetry-1.1.3-linux.tar.gz 70%
 - Downloading poetry-1.1.3-linux.tar.gz 80%
 - Downloading poetry-1.1.3-linux.tar.gz 90%
 - Downloading poetry-1.1.3-linux.tar.gz 100%

Poetry (1.1.3) is installed now. Great!
+ poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: test-functools (0.1.4)
++ poetry env info -p
+ rm -rf /root/.cache/pypoetry/virtualenvs/test-functools-HE92XDqH-py3.8
+ poetry --version
Poetry version 1.1.3
+ poetry install
Creating virtualenv test-functools-HE92XDqH-py3.8 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)
  • Installing six (1.15.0)
  • Installing attrs (20.2.0)
  • Installing iniconfig (1.1.1)
  • Installing packaging (20.4)
  • Installing pluggy (0.13.1)
  • Installing py (1.9.0)
  • Installing toml (0.10.1)
  • Installing mypy-extensions (0.4.3)
  • Installing pytest (6.1.1)
  • Installing typed-ast (1.4.1)
  • Installing typing-extensions (3.7.4.3)
  • Installing inflection (0.5.1)
  • Installing mypy (0.790)
  • Installing test-testing (0.1.4 /test-poetry-setuptools/test-testing)

Installing the current project: test-functools (0.1.4)
++ poetry env info -p
+ rm -rf /root/.cache/pypoetry/virtualenvs/test-functools-HE92XDqH-py3.8
+ git checkout poetry.lock
+ poetry install
Creating virtualenv test-functools-HE92XDqH-py3.8 in /root/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)
  • Installing six (1.15.0)
  • Installing attrs (20.2.0)
  • Installing iniconfig (1.1.1)
  • Installing packaging (20.4)
  • Installing pluggy (0.13.1)
  • Installing py (1.9.0)
  • Installing toml (0.10.1)
  • Installing mypy-extensions (0.4.3)
  • Installing pytest (6.1.1)
  • Installing typed-ast (1.4.1)
  • Installing typing-extensions (3.7.4.3)
  • Installing inflection (0.5.1)
  • Installing mypy (0.790)
  • Installing test-testing (0.1.4 /test-poetry-setuptools/test-testing)

Installing the current project: test-functools (0.1.4)