setuptools: [BUG] new release (57) breaks piptools pytest

First things first: please let me know if there is any information missing - It’s kind of hard to me to trace this down to the actual bug or to an minimum example of what is failing unfortunately.

setuptools version

setuptools==57.0.0

Python version

3.6,3.7,3.8,3.9

OS

Ubuntu, Windows, MacOS

Additional environment information

Hey, please see this issue where it was discovered: https://github.com/jazzband/pip-tools/issues/1403

First, it was wrongly attributed to be caused by pip version deprecation and feature flag, but it seems to be due to new setuptools (details in description). So currently all CI runners are broken there in that project.

• Last successful run: https://github.com/jazzband/pip-tools/actions/runs/870122483
• First failing: https://github.com/jazzband/pip-tools/actions/runs/873544759
• run which was pinned to setuptools<57 and ran through: https://github.com/jazzband/pip-tools/actions/runs/885789425

Description

Unfortunately, I’m not exactly aware of what happens behind the scenes on piptools testing suite. I’d like to share the code I could reproduce locally so far by running some of the tests manually in the fields below.

Expected behavior

There shouldn’t be anything failing because it worked on the old version before.

How to Reproduce

1. in piptools package open tests/test_cli_compile.py and copy the imports from above and the code from the function test_realistic_complex_sub_dependencies, done in the following in detail:
2. navigate to tests folder and run the following:

import os
import subprocess
import sys
from textwrap import dedent
from unittest import mock
import pytest
from pip._internal.utils.urls import path_to_url
from piptools.scripts.compile import cli
from constants import MINIMAL_WHEELS_PATH, PACKAGES_PATH
from click.testing import CliRunner

1. create runner = CliRunner(mix_stderr=False)
2. run

    wheels_dir = "wheels"
    # make a temporary wheel of a fake package
    subprocess.run(
        [
            "pip",
            "wheel",
            "--no-deps",
            "-w",
            wheels_dir,
            os.path.join(PACKAGES_PATH, "fake_with_deps", "."),
        ],
        check=True,
    )
    with open("requirements.in", "w") as req_in:
        req_in.write("fake_with_deps")  # require fake package

    out = runner.invoke(cli, ["-n", "--rebuild", "-f", wheels_dir])

which fails.

Output

expected

In [39]: out.stderr
Out[39]: '#\n# This file is autogenerated by pip-compile\n# To update, run:\n#\n#    pip-compile --find-links=wheels\n#\n--find-links wheels\n\ncolorama==0.3.9\n    # via fake-with-deps\ncornice==1.0.0\n    # via fake-with-deps\nenum34==1.1.6\n    # via fake-with-deps\nfake-with-deps==0.1\n    # via -r requirements.in\nfutures==3.0.5\n    # via fake-with-deps\ngreenlet==1.1.0\n    # via sqlalchemy\nipaddress==1.0.23\n    # via fake-with-deps\njsonschema==2.6.0\n    # via fake-with-deps\npastedeploy==2.1.1\n    # via pyramid\npyramid==1.5.8\n    # via\n    #   cornice\n    #   fake-with-deps\npython-dateutil==2.4.2\n    # via fake-with-deps\npython-memcached==1.59\n    # via fake-with-deps\npyzmq==14.7.0\n    # via fake-with-deps\nrepoze.lru==0.7\n    # via pyramid\nsimplejson==3.17.2\n    # via\n    #   cornice\n    #   fake-with-deps\nsix==1.8.0\n    # via\n    #   fake-with-deps\n    #   python-dateutil\n    #   python-memcached\nsqlalchemy==1.4.15\n    # via fake-with-deps\ntranslationstring==1.4\n    # via pyramid\nvenusian==3.0.0\n    # via pyramid\nwebob==1.8.7\n    # via pyramid\nxmltodict==0.11.0\n    # via fake-with-deps\nzope.deprecation==4.4.0\n    # via pyramid\nzope.interface==5.4.0\n    # via pyramid\n\n# The following packages are considered to be unsafe in a requirements file:\n# setuptools\nDry-run, so nothing updated.\n'

actual (failing) output

      ERROR: Command errored out with exit status 1:
       command: /Users/nalbers/git/pip-tools/thisvirtualenv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"'; __file__='"'"'/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-pip-egg-info-nkzz3eb8
           cwd: /private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/
      Complete output (29 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/__init__.py", line 18, in <module>
          from setuptools.dist import Distribution
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/_virtualenv.py", line 89, in exec_module
          old(module)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/dist.py", line 32, in <module>
          from setuptools.extern.more_itertools import unique_everseen
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
        File "<frozen importlib._bootstrap>", line 565, in module_from_spec
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 52, in create_module
          return self.load_module(spec.name)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/extern/__init__.py", line 37, in load_module
          __import__(extant)
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/__init__.py", line 1, in <module>
          from .more import *  # noqa
        File "/Users/nalbers/git/pip-tools/thisvirtualenv/lib/python3.9/site-packages/setuptools/_vendor/more_itertools/more.py", line 5, in <module>
          from concurrent.futures import ThreadPoolExecutor
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/__init__.py", line 8, in <module>
          from concurrent.futures._base import (FIRST_COMPLETED,
        File "/private/var/folders/pd/gtn_zdjj2gn1rsbkhvtckhjc0000gp/T/pip-resolver-x2nbtsp4/futures/concurrent/futures/_base.py", line 357
          raise type(self._exception), self._exception, self._traceback
                                     ^
      SyntaxError: invalid syntax
      ----------------------------------------

Code of Conduct

• I agree to follow the PSF Code of Conduct