poetry: Installing a package that depends on setuptools fails intermittently: `Backend 'setuptools.build_meta:__legacy__' is not available.`

In our CI I now retry poetry install 3 times and it works.

RUN (poetry install -vvv --no-root || poetry install -vvv --no-root || poetry install -vvv --no-root)...

I spent a couple hours looking into this, and I think I’ve identified the root cause (or at least a root cause) for this issue. I’ve set up a test repo, that fairly reliably replicates this in the github actions runner.

I believe the root cause is when multiple setup.py based dependencies are being installed from a source distribution for the first time on a system (in my example, future and restructuredtext-lint). Poetry will build both of these dependencies into a wheel in parallel. For each dependency it identifies that they depend on setuptools, checks if setuptools is in the artifact cache, finds that it isn’t, and then attempts to download a distribution for setuptools. Since there are two threads running in parallel trying to download the same file, there is a race condition where one thread succeeds and one fails (I believe this is predominately an issue for windows, with how it handles concurrent access to the file system). For the one that fails, the exception is caught, and pased up the call trace as a status return code until it is silently ignored here: installer.run(). Then, poetry proceeds to attempt to build the distribution, which fails because setuptools was never installed in the virtual environment for that dependency. I modify the executor in my github actions run so that it prints out the stack trace when exceptions are encountered, which can be seen here: Test #14. The relevant section:

Traceback (most recent call last):
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 767, in _download_link
    original_archive = self._download_archive(operation, link)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 857, in _download_archive
    with atomic_open(archive) as f:
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\contextlib.py", line 142, in __exit__
    next(self.gen)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\requests\utils.py", line 307, in atomic_open
    os.replace(tmp_name, filename)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\8d\\aa\\9e\\5ebe452d7977c93fb8e0db884b6b167954ac9814e1f25125b4552d76c8\\tmpui9rw792' -> 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\8d\\aa\\9e\\5ebe452d7977c93fb8e0db884b6b167954ac9814e1f25125b4552d76c8\\setuptools-68.2.2-py3-none-any.whl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 286, in _execute_operation
    result = self._do_execute_operation(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 387, in _do_execute_operation
    result: int = getattr(self, f"_execute_{method}")(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 512, in _execute_install
    status_code = self._install(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 550, in _install
    archive = self._download(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 754, in _download
    return self._download_link(operation, link)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 773, in _download_link
    cached_file.unlink(missing_ok=True)
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\pathlib.py", line 1[20](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:21)6, in unlink
    self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\8d\\aa\\9e\\5ebe452d7977c93fb8e0db884b6b167954ac9814e1f25125b4552d76c8\\setuptools-68.2.2-py3-none-any.whl'

Traceback (most recent call last):
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 767, in _download_link
    original_archive = self._download_archive(operation, link)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 857, in _download_archive
    with atomic_open(archive) as f:
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\contextlib.py", line 142, in __exit__
    next(self.gen)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\requests\utils.py", line 307, in atomic_open
    os.replace(tmp_name, filename)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\be\\01\\ba\\eabeb6a7ad74ee2a0542e0479545034db7365479427d5[26](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:27)7875[27](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:28)6c906\\tmpn385zi8y' -> 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\be\\01\\ba\\eabeb6a7ad74ee2a0542e0479545034db7365479427d5267875276c906\\wheel-0.41.2-py3-none-any.whl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line [28](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:29)6, in _execute_operation
    result = self._do_execute_operation(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 387, in _do_execute_operation
    result: int = getattr(self, f"_execute_{method}")(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 512, in _execute_install
    status_code = self._install(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 550, in _install
    archive = self._download(operation)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 754, in _download
    return self._download_link(operation, link)
  File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 773, in _download_link
    cached_file.unlink(missing_ok=True)
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\pathlib.py", line 1206, in unlink
    self._accessor.unlink(self)
PermissionError: [WinError [32](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:33)] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\be\\01\\ba\\eabeb6a7ad74ee2a0542e04795450[34](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:35)db7[36](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:37)5479427d5267875276c906\\wheel-0.[41](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:42).2-py3-none-any.whl'


  ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.

  at ~\.local\venv\lib\site-packages\poetry\installation\chef.py:147 in _prepare
      1[43](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:44)| 
      1[44](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:45)|                 error = ChefBuildError("\n\n".join(message_parts))
      1[45](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:46)| 
      1[46](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:47)|             if error is not None:
    > 1[47](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:48)|                 raise error from None
      1[48](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:49)| 
      1[49](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:50)|             return path
      1[50](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:51)| 
      1[51](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:52)|     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with future (0.18.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "future (==0.18.3)"'.

Which shows the errors resulting from concurrent access to the artifact cache

Note that when replicating this issue setuptools needs to not be present already in the artifact cache (which is presumably why this is mostly affecting CI, and why pre installing setuptools fixes it in some cases). poetry cache clear --all . doesn’t seem to remove it, so I’ve been manually removing the folder when I’ve been testing this.

I’m not exactly sure how this problem should be solved. The right solution probably involves some sort of locking mechanism for the artifact cache being implemented to prevent concurrent downloads of the same distribution package. The return value of installer.run should probably be checked in some manner to aid in future debugging. It could also be used in a retry loop, though that seems like a much hackier solution than locking.

Another workaround that should address the issue is:

poetry config installer.max-workers 1

since this will prevent two builds from happening in parallel (this will likely be much slower than running poetry install repeatedly though, so might not actually be of value).

It’s a different error, the key difference being ModuleNotFoundError: No module named 'distutils'. That’s because distutils is removed in Python 3.12. @scottshambaugh this is fixed in numpy 1.26.

For those experiencing this issue: update your lock files. It might’ve been luck (intermittent failure and all…), but here it started to behave again.[1] That would explain the inability to reproduce this bug as well.

[1] …and then immediately stumble into #7572, as I’m using Hypercorn, but that one is better understood and can be worked around without downgrading.

We hit this with poetry 1.6.1 installing pandas 2.0.3.

You can also try update pip, in my case it resolved the issue

@xaviernogueira I’m having the exact issue on a ‘windows-latest’ action. Switching to poetry 1.3.2 worked for me. Updating lock file did nothing.

MacOS Catalina, Python 3.10.11, Poetry 1.4.1

  • Installing promise (2.3): Failed

  ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.

Same behaviour here with scikit-learn, with Poetry 1.3.2 works fine.

that is not even close to being the same error: and by reproducing it with pip you have anyway proved that it is not a poetry problem.

I started seeing the failures in python:3.9-slim based images also. On the Azure CI VM, setup tools was already installed, so that wasn’t a viable workaround.

@robbotorigami this works so far:

poetry config installer.max-workers 1

I have encountered the same issue with poetry==1.5.1, python 3.11, but only on “windows-latest” in my GitHub actions. Works fine on “ubuntu-latest”.

I downgraded to poetry==1.3.2 which works now, but would love to be able to deploy on the latest versions. Devs please address!

Can confirm the problem with virtualenv 20.23.0 (latest at this time)

Fixed on my CI, for me the issue came from my pyproject.toml that still had the old tool.poetry.dev-dependencies section. After moving to the new tool.poetry.group.dev.dependencies style, the CI passed again.

For some libraries, change of the version of Python resolved errors.

Sorry to (possibly?) be the bearer of bad news here: This build appears to fail (on Windows) with a very similar symptom to what is described here, despite using poetry 1.7.0, which contains #8517. 🤔

  � Installing django-codemirror-widget (0.5.0)

  ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.
  
  Traceback (most recent call last):
    File "C:\Users\runneradmin\.local\venv\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\hostedtoolcache\windows\Python\3.12.0\x64\Lib\importlib\__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 994, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "C:\Users\RUNNER~1\AppData\Local\Temp\tmp5cl80_pe\.venv\Lib\site-packages\setuptools\__init__.py", line 8, in <module>
      import distutils.core
  ModuleNotFoundError: No module named 'distutils'
  

  at ~\.local\venv\Lib\site-packages\poetry\installation\chef.py:166 in _prepare
      162| 
      163|                 error = ChefBuildError("\n\n".join(message_parts))
      164| 
      165|             if error is not None:
    > 166|                 raise error from None
      167| 
      168|             return path
      169| 
      170|     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with django-codemirror-widget (0.5.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "django-codemirror-widget (==0.5.0)"'.

Notably, the suggested command succeeds:

pip wheel --no-cache-dir --use-pep517 "django-codemirror-widget (==0.5.0)"                                                                                               andreas@arc 15:25
/home/andreas/src/env-3.12/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py:111: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pip._vendor.pkg_resources import find_distributions
Collecting django-codemirror-widget==0.5.0
  Downloading django-codemirror-widget-0.5.0.tar.gz (6.6 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: django-codemirror-widget
  Building wheel for django-codemirror-widget (pyproject.toml) ... done
  Created wheel for django-codemirror-widget: filename=django_codemirror_widget-0.5.0-py3-none-any.whl size=7918 sha256=994b804c1d4228457b59ce78a09d2be116f56aeba47a5c968504343582afb26c
  Stored in directory: /tmp/pip-ephem-wheel-cache-1cu5q9ol/wheels/35/bd/a5/48fb66e57f3161108f210d9dbcb36732bf7afcd6e63a4f38da
/home/andreas/src/env-3.12/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py:73: DeprecationWarning: Unimplemented abstract methods {'locate_file'}
  return cls(files, info_location)
Successfully built django-codemirror-widget

Thanks to @robbotorigami (https://github.com/python-poetry/poetry/issues/7611#issuecomment-1747836233) I was able to reproduce the issue and provide a fix in #8517. Feel free to try the PR.

I tried every other option presented here, all failed.

What striked me is that I have several stages in my gitlab CI, all with the very same poetry install, all were working and just one failed at some point recently.

The only difference is the image used : so I switched from image: python that was the only stage failing in my CI to image: python:3.11 and it is suddently working again.

I was having this issue in my GitHub Actions Windows CI Pipeline as well. Without downgrading poetry, I could workaround the missing backend by ensuring setuptools was installed in poetry’s environment, but YMMV.

For a pipx install of poetry, this worked for me:

pipx install poetry==1.6.1
pipx inject poetry setuptools==68.1.2

And this should be a more general fix:

poetry self add setuptools

Same issue running in github-actions with: windows-latest, poetry 1.6.1, virtualenv 20.24.4, python 3.10.11. Happens during installation of pybloom-live (4.0.0) or fire (0.5.0). Interestingly, it never failed for fire before pybloom-live was added, as if only the number of deps mattered.

Downgrading to 1.3.2 or repeating the poetry install command both work as workarounds.

I’m running into the same error when using the windows-latest runner for Github Actions. The CI works fine for py3.10 and py3.11, but fails for py3.8, 3.9 with the same error:

ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.

  at C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\poetry\installation\chef.py:147 in _prepare
      143| 
      144|                 error = ChefBuildError("\n\n".join(message_parts))
      145| 
      146|             if error is not None:
    > 147|                 raise error from None
      148| 
      149|             return path
      150| 
      151|     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with olefile (0.46) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "olefile (==0.46)"'.


  ChefBuildError

  Backend 'setuptools.build_meta:__legacy__' is not available.

  at C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\poetry\installation\chef.py:147 in _prepare
      143| 
      144|                 error = ChefBuildError("\n\n".join(message_parts))
      145| 
      146|             if error is not None:
    > 147|                 raise error from None
      148| 
      149|             return path
      150| 
      151|     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with compressed-rtf (1.0.6) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "compressed-rtf (==1.0.6)"'.

EDIT: Update pytest to latest version and now py3.8, 3.10, 3.11 fail with the same error but py3.9 passes…

For those experiencing this issue: update your lock files. It might’ve been luck (intermittent failure and all…), but here it started to behave again.[1] That would explain the inability to reproduce this bug as well.

[1] …and then immediately stumble into #7572, as I’m using Hypercorn, but that one is better understood and can be worked around without downgrading.

Updating the lock files solved for me, thanks @gpongelli!

We encounter this sporadically with Poetry 1.4.1 running in Windows container as well. The error message is like:

 • Installing wrapt (1.14.1)
  ChefBuildError
  Backend 'setuptools.build_meta:__legacy__' is not available.
  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with atomicwrites (1.4.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "atomicwrites (==1.4.1)"'.