pip: Unable to upgrade setuptools with adopted distutils

Environment

• pip version: 20.2.2
• Python version: 3.8.5
• OS: macOS 10.15.6

Description

In pypa/setuptools#417, Setuptools is seeking to adopt distutils. With the current release (49.6), Setuptools has a vendored copy of distutils, but that copy is not activated except when SETUPTOOLS_USE_DISTUTILS=local in the environment.

In pypa/setuptools#2259, the team identified the need to make the distutils replacement more aggressive so that packages that import distutils before import setuptools would get the version from Setuptools (and not some unfortunate mix). This behavior was released in Setuptools 49.3, but still dormant unless SETUPTOOLS_USE_DISTUTILS=local.

In pypa/setuptools#2255, we’re seeking to make that behavior the default (opt-out with SETUPTOOLS_USE_DISTUTILS=stdlib), but the test suite caught an undesirable behavior when attempting to upgrade setuptools–it seems that there’s a race condition in pip when the distutils that setuptools is supplying disappears during an upgrade.

Expected behavior

pip should be able to upgrade setuptools even when setuptools supplies distutils.

How to Reproduce

1. Install setuptools 49.3 or later.
2. Install any other version of setuptools with SETUPTOOLS_USE_DISTUTILS=local set.

Output

draft $ virtualenv env --no-setuptools
created virtual environment CPython3.8.5.final.0-64 in 271ms
  creator CPython3Posix(dest=/Users/jaraco/draft/env, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, wheel=bundle, via=copy, app_data_dir=/Users/jaraco/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
draft $ env/bin/pip install 'setuptools<49.6'
Collecting setuptools<49.6
  Using cached setuptools-49.5.0-py3-none-any.whl (803 kB)
Installing collected packages: setuptools
Successfully installed setuptools-49.5.0
draft $ env SETUPTOOLS_USE_DISTUTILS=local env/bin/pip install setuptools==49.6
Collecting setuptools==49.6
  Using cached setuptools-49.6.0-py3-none-any.whl (803 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.5.0
    Uninstalling setuptools-49.5.0:
      Successfully uninstalled setuptools-49.5.0
  Rolling back uninstall of setuptools
  Moving to /Users/jaraco/draft/env/bin/easy_install
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-0v5mjgxu/easy_install
  Moving to /Users/jaraco/draft/env/bin/easy_install-3.8
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-0v5mjgxu/easy_install-3.8
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/__pycache__/easy_install.cpython-38.pyc
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-jg6byxv1/easy_install.cpython-38.pyc
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/_distutils_hack/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~distutils_hack
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/distutils-precedence.pth
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-te4egp43/distutils-precedence.pth
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/easy_install.py
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-te4egp43/easy_install.py
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/pkg_resources/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~kg_resources
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools-49.5.0.dist-info/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~etuptools-49.5.0.dist-info
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~etuptools
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 412, in run
    installed = install_given_reqs(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 82, in install_given_reqs
    requirement.install(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 778, in install
    scheme = get_scheme(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 185, in get_scheme
    scheme = distutils_scheme(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 125, in distutils_scheme
    i.finalize_options()
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/command/install.py", line 381, in finalize_options
    self.set_undefined_options('build',
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
    src_cmd_obj = self.distribution.get_command_obj(src_cmd)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 858, in get_command_obj
    klass = self.get_command_class(command)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/dist.py", line 771, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build'