setuptools: ModuleNotFoundError: No module named 'setuptools._distutils'

pip install . suddenly started failing for many packages. Since setuptools just got a new version and pip didn’t, and setuptools appears in the error, I’m guessing it’s related to setuptools 50. Apologies if this turns out to be wrong.

This can be seen in any number of repositories, such as https://github.com/dHannasch/tox-sitepackages-example.

This doesn’t appear to be quite the same as https://github.com/pypa/setuptools/issues/2352. I don’t fully understand all the implications of https://github.com/pypa/setuptools/issues/2350, but SETUPTOOLS_USE_DISTUTILS=stdlib has no effect on this (ditto SETUPTOOLS_USE_DISTUTILS=1), so I think this is a separate issue.

Travis log:

$ python -m pip install .

Processing /home/travis/build/dHannasch/tox-sitepackages-example

  Installing build dependencies ... \/done

  Getting requirements to build wheel ... done

ERROR: Exception:

Traceback (most recent call last):

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 216, in _main

    status = self.run(options, args)

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper

    return func(self, options, args)

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 325, in run

    reqs, check_supported_wheels=not options.target_dir

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve

    discovered_reqs.extend(self._resolve_one(requirement_set, req))

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one

    abstract_dist = self._get_abstract_dist_for(req_to_install)

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for

    abstract_dist = self.preparer.prepare_linked_requirement(req)

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement

    req, self.req_tracker, self.finder, self.build_isolation,

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution

    abstract_dist.prepare_distribution_metadata(finder, build_isolation)

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata

    self._setup_isolation(finder)

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation

    reqs = backend.get_requires_for_build_wheel()

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel

    'config_settings': config_settings

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook

    raise BackendUnavailable(data.get('traceback', ''))

pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):

  File "/home/travis/build/dHannasch/tox-sitepackages-example/py38/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend

    obj = import_module(mod_path)

  File "/opt/python/3.7.1/lib/python3.7/importlib/__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

  File "<frozen importlib._bootstrap>", line 983, in _find_and_load

  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import

  File "<frozen importlib._bootstrap>", line 983, in _find_and_load

  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 728, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/opt/python/3.7.1/lib/python3.7/site-packages/setuptools/__init__.py", line 5, in <module>

    import distutils.core

  File "/tmp/pip-build-env-co0toouh/overlay/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 82, in create_module

    return importlib.import_module('._distutils', 'setuptools')

  File "/opt/python/3.7.1/lib/python3.7/importlib/__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

ModuleNotFoundError: No module named 'setuptools._distutils'

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 44
  • Comments: 21 (5 by maintainers)

Commits related to this issue

Most upvoted comments

@shakthifuture Setting environment variable SETUPTOOLS_USE_DISTUTILS=stdlib is a workaround, e.g.:

$ export SETUPTOOLS_USE_DISTUTILS=stdlib
$ pip3 install […]
+54
simmel on Sep 1, 2020

So, setuptools 50.0 is fundamentally broken and breaks the entire Python ecosystem – both open-source and not. Well, isn’t that special. Heads need rolling (especially those currently attached to the still-functioning torsos of managerial project leads). Until the community tastes sweet vengeance, the following is a slightly saner solution than @dHannasch’s excellent starting point:

[build-system]
requires = [
    "setuptools!=50.0",
    "wheel",
]

That is to say, downstream projects should probably only blacklist the specific version of setuptools known to catastrophically fail under the fairly safe assumption that the next stable release will either hopefully revert or perhaps even correctly fix the breakage.

I can confirm the above circumvention behaves as expected in a project just maliciously blind-sided by this packaging horror show. Passing tox-based tests or it didn’t happen, of course.

Die, setuptools 50.0! Die! 🩸

+22
leycec on Aug 31, 2020

OK - I’m in no way an expert on PEP-517, but I think the bug here is in pip, not setuptools - or maybe there’s one bug in each.

If a pyproject.toml contains something like:

[build-system]
requires = ["setuptools"]

then pip installs the latest version of setuptools into a temp directory with a name like (for instance): /tmp/pip-build-env-5v90m1w9/overlay/lib/python3.8/site-packages

After it does that, it eventually calls the get_requires_for_build_wheel hook, in a new subprocess. In that subprocess, if pip was run inside a virtualenv created with --system-site-packages, then sys.path is something like:

['/home/matt/some_venv/lib/python3.8/site-packages/pip/_vendor/pep517',
 '/tmp/pip-build-env-5v90m1w9/site',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/usr/local/lib/python3.8/dist-packages',
 '/usr/lib/python3/dist-packages',
 '/tmp/pip-build-env-5v90m1w9/overlay/lib/python3.8/site-packages',
 '/tmp/pip-build-env-5v90m1w9/normal/lib/python3.8/site-packages']

The overlay directory is after the system site packages directory, which means that any build dependencies installed by pip because they were required by the pyproject.toml are later in the search path than the ones that were already installed in the site-packages directory.

I think there’s one bug in each here: the _distutils_hack probably shouldn’t do anything if there’s a different setuptools ahead of it in sys.path - whichever setuptools is earliest in the module search path should be completely responsible for everything; a .pth file from later in the search path shouldn’t be screwing it up. And, pip probably shouldn’t be calling the hook in an environment where already-installed modules are taking precedence over modules specifically required by the pyproject.toml - the whole point of an isolated build is to isolate you from what’s installed system-wide.

+13
godlygeek on Sep 2, 2020

I had same issue. Removing pyproject.toml from the library which failed to install pass through it

+12
vkrot-exos on Aug 31, 2020

FYI, the AWS lambdas running Python were also broken.

https://stackoverflow.com/questions/63688774/some-aws-lambda-functions-stopped-working-with-no-module-named-setuptools-dist

+4
canassa on Sep 2, 2020

Yeah, we’ve had a suspicion about the pip as well. We then tried downgrading to v18.1 and it worked here!

+2
ribaldorafael on Sep 2, 2020

This Dockerfile is somewhat convoluted, but this is the simplest way I’ve found to reproduce this problem so far:

FROM ubuntu:20.04
WORKDIR /root
RUN apt-get update && \
    apt-get install python3-dev python3-setuptools python3-venv python3-wheel gcc \
                    libssl-dev libffi-dev -y && \
    python3 -m venv --system-site-packages venv
ENV PATH /root/venv/bin:$PATH
RUN pip install --no-binary=:all: cryptography

EDIT: The installation succeeds if you set SETUPTOOLS_USE_DISTUTILS=stdlib

+2
bmw on Sep 1, 2020

That’s true, and a useful workaround, but I think that’s just because with no pyproject.toml, setuptools 50 doesn’t get installed in the first place.

So anyone can avoid this problem for now by specifying setuptools<50 in their pyproject.toml, but the fact that that works probably isn’t relevant for debugging setuptools, sadly. Oh well.

E.g.

[build-system]
requires = [
    "setuptools>=30.3.0,<50",
    "wheel",
    "pytest-runner",
    "setuptools_scm>=3.3.1",
]
+2
dHannasch on Aug 31, 2020

There is, in fact, an old pip issue for exactly this bug. https://github.com/pypa/pip/issues/6264#issuecomment-685230919.

+1
godlygeek on Sep 2, 2020

I’ve been able to consistently reproduce this with a virtualenv created with --system-site-packages where setuptools < 50 is installed in the system site packages directory. I filed https://github.com/pypa/virtualenv/issues/1934 for it ~because the behavior differs between virtualenv and venv~. Update: Scratch that, after seeing https://github.com/pypa/setuptools/issues/2353#issuecomment-685177014 I tried reproducing it with venv instead of virtualenv again and succeeded, so it’s not in any way specific to virtualenv as opposed to venv. In any event the reproduction test case from that issue should still be useful.

Clearly there is some case where an isolated build isn’t being properly isolated, and it’s managing to pick up a different setuptools than the one that was installed for the isolated build.

+1
godlygeek on Sep 1, 2020
Read more comments on GitHub
← setuptools: `InvalidVersion` exception when invalid version used on Setuptools 66
setuptools: Newlines in the `description` field produce a malformed PKG-INFO →