poetry: Installing a package that depends on setuptools fails intermittently: `Backend 'setuptools.build_meta:__legacy__' is not available.`
- Poetry version: 1.4.0
- Python version: 3.10.9
- OS version and name: Microsoft Windows Server 2022 10.0.20348 Datacenter (
windows-2022
GitHub runner)
- I am on the latest stable Poetry version, installed using a recommended method.
- I have searched the issues of this repo and believe that this is not a duplicate.
- I have consulted the FAQ and blog for any relevant entries or release notes.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvv
option) and have included the output below.
Issue
Installing a package that depends on setuptools fails intermittently. This bug does not occur in poetry 1.3 and is found in poetry 1.4.
ChefBuildError
Backend 'setuptools.build_meta:__legacy__' is not available.
at C:\Program Files (x86)\pipx\venvs\poetry\lib\site-packages\poetry\installation\chef.py:152 in _prepare
148|
149| error = ChefBuildError("\n\n".join(message_parts))
150|
151| if error is not None:
> 152| raise error from None
153|
154| return path
155|
156| def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with docopt (0.6.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "docopt (==0.6.2)"'.
View full logs
$ poetry install
Installing dependencies from lock file
Package operations: 150 installs, 2 updates, 0 removals
Installing certifi (2022.12.7)
Installing charset-normalizer (2.0.12)
Installing idna (3.4)
Installing pyasn1 (0.4.8)
Installing urllib3 (1.26.14)
Installing cachetools (5.3.0)
Installing oauthlib (3.2.2)
Installing pyasn1-modules (0.2.8)
Installing requests (2.28.2)
Installing six (1.16.0)
Installing rsa (4.9)
Installing catalogue (2.0.8)
Installing colorama (0.4.6)
Installing google-auth (2.16.0)
Installing markupsafe (2.1.2)
Installing requests-oauthlib (1.3.1)
Installing multidict (5.2.0)
Installing typing-extensions (4.4.0)
Installing tzdata (2022.7)
Installing absl-py (1.3.0)
Installing click (8.1.3)
Installing cymem (2.0.7)
Installing frozenlist (1.3.3)
Installing grpcio (1.51.1)
Installing google-auth-oauthlib (0.4.6)
Installing locket (1.0.0)
Installing markdown (3.4.1)
Installing murmurhash (1.0.9)
Installing numpy (1.23.5)
Installing pamqp (3.2.1)
Installing protobuf (3.19.6)
Installing pydantic (1.10.4)
Installing pyparsing (3.0.9)
Installing pyreadline3 (3.4.1)
Installing pytz-deprecation-shim (0.1.0.post0)
Installing ruamel-yaml-clib (0.2.7)
Updating setuptools (65.3.0 -> 66.1.1)
Installing srsly (2.4.5)
Installing tensorboard-data-server (0.6.1)
Installing tensorboard-plugin-wit (1.8.1)
Installing termcolor (2.2.0)
Installing toolz (0.12.0)
Installing werkzeug (2.2.2)
Updating wheel (0.37.1 -> 0.38.4)
Installing yarl (1.8.2)
Installing aiormq (6.6.4)
Installing aiosignal (1.3.1)
Installing astunparse (1.6.3)
Installing async-timeout (4.0.2)
Installing attrs (22.1.0)
Installing blis (0.7.9)
Installing cloudpickle (2.2.1)
Installing confection (0.0.4)
Installing docopt (0.6.2)
Installing fire (0.5.0)
Installing flatbuffers (23.1.21)
Installing fsspec (2023.1.0)
Installing gast (0.5.3)
Installing google-pasta (0.2.0)
Installing greenlet (2.0.1)
Installing h5py (3.8.0)
Installing humanfriendly (10.0)
Installing joblib (1.2.0)
Installing keras (2.8.0)
Installing keras-preprocessing (1.1.2)
Installing libclang (15.0.6.1)
Installing msgpack (1.0.4)
Installing opt-einsum (3.3.0)
Installing packaging (20.9)
Installing partd (1.3.0)
Installing preshed (3.0.8)
Installing pyrsistent (0.19.3)
Installing python-crfsuite (0.9.9)
Installing python-dateutil (2.8.2)
Installing pytz (2022.1)
Installing pyyaml (6.0)
Installing ruamel-yaml (0.17.21)
Installing scipy (1.8.1)
Installing smart-open (6.3.0)
Installing tabulate (0.9.0)
Installing tensorboard (2.8.0)
Installing tensorflow-estimator (2.8.0)
Installing tensorflow-io-gcs-filesystem (0.30.0)
Installing threadpoolctl (3.1.0)
Installing tqdm (4.64.1)
Installing typeguard (2.13.3)
Installing typer (0.7.0)
Installing tzlocal (4.2)
Installing wasabi (0.10.1)
Installing wrapt (1.14.1)
ChefBuildError
Backend 'setuptools.build_meta:__legacy__' is not available.
at C:\Program Files (x86)\pipx\venvs\poetry\lib\site-packages\poetry\installation\chef.py:152 in _prepare
148|
149| error = ChefBuildError("\n\n".join(message_parts))
150|
151| if error is not None:
> 152| raise error from None
153|
154| return path
155|
156| def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with docopt (0.6.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "docopt (==0.6.2)"'.
Interim Solutions
There are several workarounds below that mitigate this issue.
- Downgrade poetry version to 1.3
pipx install "poetry>=1.3.0,<1.4.0"
- Try
poetry install
several times
poetry install -vvv --no-root || poetry install -vvv --no-root || poetry install -vvv --no-root...
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 46
- Comments: 66 (14 by maintainers)
Commits related to this issue
- chore(ci): add info to help resolution of python-poetry/poetry#7611 — committed to Kosmorro/lib by Deuchnord a year ago
- deactivate windows tests (temporarily) setup-tools error: https://github.com/python-poetry/poetry/issues/7611 — committed to CoLRev-Environment/colrev by geritwagner a year ago
- deactivate windows tests (temporarily) setup-tools error: https://github.com/python-poetry/poetry/issues/7611 — committed to CoLRev-Environment/colrev by geritwagner a year ago
- Refresh poetry lock The older releases in the lock can cause the CI build to fail with more recent Poetry versions, see python-poetry/poetry#7611. Updating the lock should remedy this. — committed to mjpieters/fastapi-cache by mjpieters a year ago
- chore(ci): fix CI (#64) * chore(ci): add info to help resolution of python-poetry/poetry#7611 * Fix pyproject config --------- Co-authored-by: Jérôme Deuchnord <jerome@deuchnord.fr> — committed to Kosmorro/lib by Deuchnord a year ago
- chore(ci): fix CI (#64) * chore(ci): add info to help resolution of python-poetry/poetry#7611 * Fix pyproject config --------- Co-authored-by: Jérôme Deuchnord <jerome@deuchnord.fr> — committed to Kosmorro/lib by Deuchnord a year ago
- Updates to get CI working with no errors (#12) Fix intermittent CI issues with pypy-3.8 on windows latest due to issue https://github.com/python-poetry/poetry/issues/7611 being encountered when tryin... — committed to sdb9696/python-kasa by sdb9696 a year ago
- ci: :green_heart: Downgrading poetry to try and fix bug (see description) https://github.com/python-poetry/poetry/issues/7611 — committed to xaviernogueira/xpublish-file-metadata by xaviernogueira a year ago
- ci(github-acitons): pin poetry version to 1.3.2 due to https://github.com/python-poetry/poetry/issues/7611 — committed to davidlday by Lee-W 10 months ago
- ci(github-acitons): pin poetry version to 1.3.2 due to https://github.com/python-poetry/poetry/issues/7611 — committed to by Lee-W 10 months ago
- build: :pushpin: Pin to version `1.5.1` for Windows See also: https://github.com/python-poetry/poetry/issues/7611 Plus update types — committed to Anselmoo/spectrafit by Anselmoo 10 months ago
- build: :pushpin: Pin to version `1.5.1` for Windows See also: https://github.com/python-poetry/poetry/issues/7611 Plus update types — committed to Anselmoo/spectrafit by Anselmoo 10 months ago
- docs: :dizzy: Update color schema (#986) * build(deps): Bump python-pptx from 0.6.21 to 0.6.22 Bumps [python-pptx](https://github.com/scanny/python-pptx) from 0.6.21 to 0.6.22. - [Changelog](http... — committed to Anselmoo/spectrafit by Anselmoo 10 months ago
- Fix(InstallDeps): Install poetry < 1.4 to avoid https://github.com/python-poetry/poetry/issues/7611 — committed to nextcloud/llm by marcelklehr 9 months ago
- Fix(InstallDeps): Install poetry < 1.4 to avoid https://github.com/python-poetry/poetry/issues/7611 — committed to nextcloud/llm by marcelklehr 9 months ago
- poetry builds: try to fix poetry build flakiness https://github.com/python-poetry/poetry/issues/7611#issuecomment-1680461513 https://github.com/python-poetry/poetry/issues/7611#issuecomment-171144353... — committed to teamlumos/base-images by millimat 9 months ago
- ci: use Poetry 1.3.2 to avoid intermittent failure Workaround for the following issue: https://github.com/python-poetry/poetry/issues/7611 — committed to unioslo/tsd-file-api by haatveit 9 months ago
- [internal] poetry install brach mit ChefBuildError ab Workaround für Fehler in https://github.com/baltech-ag/bec2format/actions/runs/6419938109/job/17430890773 Siehe https://github.com/python-poetry... — committed to baltech-ag/bec2format by jkatins 9 months ago
- [internal] poetry install brach mit ChefBuildError ab Workaround für Fehler in https://github.com/baltech-ag/bec2format/actions/runs/6419938109/job/17430890773 Siehe https://github.com/python-poetry... — committed to baltech-ag/bec2format by jkatins 9 months ago
- CHORE: Test workaround for Poetry bug - Got an issue while installing project dependencies on Python 3.12 on CI. Workaround proposed in https://github.com/python-poetry/poetry/issues/7611. — committed to gird-dev/gird by truhanen 8 months ago
In our CI I now retry
poetry install
3 times and it works.I spent a couple hours looking into this, and I think I’ve identified the root cause (or at least a root cause) for this issue. I’ve set up a test repo, that fairly reliably replicates this in the github actions runner.
I believe the root cause is when multiple
setup.py
based dependencies are being installed from a source distribution for the first time on a system (in my example,future
andrestructuredtext-lint
). Poetry will build both of these dependencies into a wheel in parallel. For each dependency it identifies that they depend on setuptools, checks if setuptools is in the artifact cache, finds that it isn’t, and then attempts to download a distribution for setuptools. Since there are two threads running in parallel trying to download the same file, there is a race condition where one thread succeeds and one fails (I believe this is predominately an issue for windows, with how it handles concurrent access to the file system). For the one that fails, the exception is caught, and pased up the call trace as a status return code until it is silently ignored here: installer.run(). Then, poetry proceeds to attempt to build the distribution, which fails because setuptools was never installed in the virtual environment for that dependency. I modify the executor in my github actions run so that it prints out the stack trace when exceptions are encountered, which can be seen here: Test #14. The relevant section:Which shows the errors resulting from concurrent access to the artifact cache
Note that when replicating this issue setuptools needs to not be present already in the artifact cache (which is presumably why this is mostly affecting CI, and why pre installing setuptools fixes it in some cases).
poetry cache clear --all .
doesn’t seem to remove it, so I’ve been manually removing the folder when I’ve been testing this.I’m not exactly sure how this problem should be solved. The right solution probably involves some sort of locking mechanism for the artifact cache being implemented to prevent concurrent downloads of the same distribution package. The return value of
installer.run
should probably be checked in some manner to aid in future debugging. It could also be used in a retry loop, though that seems like a much hackier solution than locking.Another workaround that should address the issue is:
since this will prevent two builds from happening in parallel (this will likely be much slower than running
poetry install
repeatedly though, so might not actually be of value).It’s a different error, the key difference being
ModuleNotFoundError: No module named 'distutils'
. That’s because distutils is removed in Python 3.12. @scottshambaugh this is fixed in numpy 1.26.For those experiencing this issue: update your lock files. It might’ve been luck (intermittent failure and all…), but here it started to behave again.[1] That would explain the inability to reproduce this bug as well.
[1] …and then immediately stumble into #7572, as I’m using Hypercorn, but that one is better understood and can be worked around without downgrading.
We hit this with poetry 1.6.1 installing pandas 2.0.3.
You can also try update pip, in my case it resolved the issue
@xaviernogueira I’m having the exact issue on a ‘windows-latest’ action. Switching to
poetry
1.3.2 worked for me. Updating lock file did nothing.MacOS Catalina, Python 3.10.11, Poetry 1.4.1
Same behaviour here with
scikit-learn
, with Poetry 1.3.2 works fine.that is not even close to being the same error: and by reproducing it with pip you have anyway proved that it is not a poetry problem.
I started seeing the failures in
python:3.9-slim
based images also. On the Azure CI VM, setup tools was already installed, so that wasn’t a viable workaround.@robbotorigami this works so far:
I have encountered the same issue with poetry==1.5.1, python 3.11, but only on “windows-latest” in my GitHub actions. Works fine on “ubuntu-latest”.
I downgraded to poetry==1.3.2 which works now, but would love to be able to deploy on the latest versions. Devs please address!
Can confirm the problem with virtualenv 20.23.0 (latest at this time)
Fixed on my CI, for me the issue came from my
pyproject.toml
that still had the oldtool.poetry.dev-dependencies
section. After moving to the newtool.poetry.group.dev.dependencies
style, the CI passed again.For some libraries, change of the version of Python resolved errors.
Sorry to (possibly?) be the bearer of bad news here: This build appears to fail (on Windows) with a very similar symptom to what is described here, despite using poetry 1.7.0, which contains #8517. 🤔
Notably, the suggested command succeeds:
Thanks to @robbotorigami (https://github.com/python-poetry/poetry/issues/7611#issuecomment-1747836233) I was able to reproduce the issue and provide a fix in #8517. Feel free to try the PR.
I tried every other option presented here, all failed.
What striked me is that I have several stages in my gitlab CI, all with the very same poetry install, all were working and just one failed at some point recently.
The only difference is the image used : so I switched from
image: python
that was the only stage failing in my CI toimage: python:3.11
and it is suddently working again.I was having this issue in my GitHub Actions Windows CI Pipeline as well. Without downgrading poetry, I could workaround the missing backend by ensuring
setuptools
was installed in poetry’s environment, but YMMV.For a
pipx
install ofpoetry
, this worked for me:And this should be a more general fix:
Same issue running in github-actions with: windows-latest, poetry 1.6.1, virtualenv 20.24.4, python 3.10.11. Happens during installation of pybloom-live (4.0.0) or fire (0.5.0). Interestingly, it never failed for fire before pybloom-live was added, as if only the number of deps mattered.
Downgrading to 1.3.2 or repeating the
poetry install
command both work as workarounds.I’m running into the same error when using the
windows-latest
runner for Github Actions. The CI works fine for py3.10 and py3.11, but fails for py3.8, 3.9 with the same error:EDIT: Update pytest to latest version and now py3.8, 3.10, 3.11 fail with the same error but py3.9 passes…
Updating the
lock
files solved for me, thanks @gpongelli!We encounter this sporadically with Poetry 1.4.1 running in Windows container as well. The error message is like: