pip: virtualenv with --system-site-packages breaks pip's build isolation

I should also mention that --no-use-pep517 fixes the problem for me.

I believe this is the root cause of #8823 and https://github.com/pypa/setuptools/issues/2353 - a PEP 517 build of a package is performed, its build requirements are installed into a directory that is added to sys.path for the subprocess that runs the build backend, but they are put later in sys.path than an older version of setuptools. There’s a .pth file in the latest setuptools that tries to do some magic, and the magic that it does is busted because the version of setuptools that’s earlier on the module search path is an earlier version, incompatible with the .pth file.

A simpler reproducer of this than what I see above is:

python3.8 -m venv --system-site-packages venv38
./venv/bin/pip install --no-binary :all: --ignore-installed python-dateutil

which fails with:

Collecting python-dateutil
  Using cached python-dateutil-2.8.1.tar.gz (331 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: /home/matt/venv/bin/python /home/matt/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7fsbsn8s/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm
       cwd: None
  Complete output (60 lines):
  Ignoring setuptools: markers 'python_version == "3.3"' don't match your environment
  Collecting setuptools
    Using cached setuptools-50.0.2.zip (2.2 MB)
  Collecting wheel
    Using cached wheel-0.35.1.tar.gz (59 kB)
  Collecting setuptools_scm
    Using cached setuptools_scm-4.1.2.tar.gz (48 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Exception:
  Traceback (most recent call last):
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
      status = self.run(options, args)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 325, in run
      reqs, check_supported_wheels=not options.target_dir
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement
      req, self.req_tracker, self.finder, self.build_isolation,
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 91, in _get_prepared_distribution
      abstract_dist.prepare_distribution_metadata(finder, build_isolation)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
      self._setup_isolation(finder)
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_internal/distributions/sdist.py", line 96, in _setup_isolation
      reqs = backend.get_requires_for_build_wheel()
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 161, in get_requires_for_build_wheel
      'config_settings': config_settings
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/wrappers.py", line 265, in _call_hook
      raise BackendUnavailable(data.get('traceback', ''))
  pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
    File "/home/matt/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 86, in _build_backend
      obj = import_module(mod_path)
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 994, in _gcd_import
    File "<frozen importlib._bootstrap>", line 971, in _find_and_load
    File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 678, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 5, in <module>
      import distutils.core
    File "/tmp/pip-build-env-zel77eqn/overlay/lib/python3.6/site-packages/_distutils_hack/__init__.py", line 83, in create_module
      return importlib.import_module('setuptools._distutils')
    File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ModuleNotFoundError: No module named 'setuptools._distutils'

  ----------------------------------------
ERROR: Command errored out with exit status 2: /home/matt/venv/bin/python /home/matt/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-7fsbsn8s/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools; python_version != '"'"'3.3'"'"'' 'setuptools<40.0; python_version == '"'"'3.3'"'"'' wheel setuptools_scm Check the logs for full command output.

This failure is happening because there are two versions of setuptools on the search path, an older one (from the site-packages directory of the interpreter that created the virtualenv) and a newer one (installed as part of the build requirements for the PEP 517 build). The newer one is shipped along with a .pth file that tries to import setuptools._distutils. The .pth file runs even though another version of setuptools is before it in the module search path, but the import it performs fails because the old version, earlier on the search path, doesn’t contain setuptools._distutils.

That’s exactly my plan 😛 pip already uses pep517 under the hood, and the main reason it does not also use its isolated environment implementation is because pep517 maintainers don’t want anything outside of the core PEP 517 interface to be depended on. It would be best IMO if the isolated environment implementation is maintained outside of pip (so other projects can reuse it), and python-build would be in the perfect position since it does the exactly thing pip needs, and already plans to join PyPA.

venv --without-pip installs python.exe, pythonw.exe, pyvenv.cfg and 4 activaion scripts on my Windows PC, and takes essentially no time. I think it’s perfectly acceptable. It’s not like the “old days” when creating a virtual environment installed a big chunk of the stdlib.

If you can’t use virtualenv then I’ve done something wrong with it

I don’t think it’s that we can’t use virtualenv, but rather (if we’re not installing pip etc) that we don’t need it - venv --without-pip is perfectly sufficient, and comes with the stdlib, so there are no vendoring issues.

I also hit this problem. In my case, downgrade pip. it works.

pip install --upgrade pip==19.0
pip install numpy

@pfmoore if virtualenv inherits the system site package (and that has a setuptoools before 40.8.0) the pip install of the setuptools build dependency is ignored (as pip does not check the version of what is installed unless -U is passed, and says any setuptools satisfies the install requirement); therefore once pip tries to get the build backend the above failure is thrown. I don’t think there needs to be anything wrong with system python to fall into this issue. pip should make sure to never-ever create an isolated build environment that inherits from the system site package, which is not the case at the moment.

Inside a Debian GNU/Linux 10 (buster) Docker container we create a virtual-env with --system-site-packages Inside we have pip == 19.2.1 We are using pyproject.toml, requirements.txt and setup.cfg and install the latest setuptools == 63.2.0 using the pyproject.toml

We have the following code in the setup.cfg:

[options]
packages = find:
install_requires = file: requirements.txt

Now we hit the error where the get_requires_for_build_wheel fails with parsing error of the above setup.cfg.

File "/usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
      requirement_string[e.loc:e.loc + 8]))
      pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "': requir'"

Upon inspection we found out that pip instead of loading the latest setuptools from the build environment loaded the setuptools == 40.8.0 from dist-packages and used the build_meta from /usr/lib/python3/dist-packages/setuptools/build_meta.py

For those struggling with same bug using pip and venv on Debian distros, the current workaround we are using is to either uninstall the python3-setuptools or try to use the --no-build-isolation flag with pip

It’s still worth having the reproducer, when we find out the cause of the main issue here we can check if it addresses the debian case as well.

I think the cause is clear. pip implements build isolation by removing the current platlib and purelib directories (and any directories from .pth files in those directories) from sys.path using the code here. Therefore, if there are any system site directories that are not covered by that, they remain in sys.path in the build environment.

• In the case of a virtual environment with --system-site-packages, the current platlib/purelib directories are the ones inside the venv, so any system site directories are not removed. I expect this to happen on any distribution.

• In the case of Debian, the system Python has an extra site directory, /usr/local/lib/python3.6/dist-packages, which is neither the platlib nor purelib directory, so it’s not removed.

It seems to me that the simplest fix would be to remove all directories in site.getsitepackages() from sys.path and not just platlib/purelib.

@pganssle it looks like it’s time to remove --system-site-packages from the title. I’ve already hit two instances of the bug with --no-site-packages.

FTR I’m hitting this with tox + system-site-packages with setuptools>=40.8.0 in pyproject.toml under Python 2.7.15. I was actually hitting pypa/setuptools#1136 originally (which I knew was fixed a while back) and only after some time of playing around with removing pyproject.toml and then just playing with --no-build-isolation I’ve found this one…

Couldn’t figure it out for a while because it was only reproducible in Travis CI which drops you into a virtualenv from the beginning automatically. Had to request debug access to their VMs to nail it down…

@pfmoore Same problem when pip and setuptools were installed with pip, when using latest cryptography from PyPI. Note that there’s no other Python 3 installation on the system (and hence no other setuptools either).

$ which python
/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug
$ curl -OL https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  480k  100  480k    0     0   755k      0 --:--:-- --:--:-- --:--:--  754k

$ tar xfz cryptography-2.6.1.tar.gz
$ cd cryptography-2.6.1

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
    Preparing wheel metadata ... error
    Complete output from command /prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_:
    Traceback (most recent call last):
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_" failed with error code 1 in /tmp/pip-req-build-s_w63g1g

Works fine when using --no-use-pep517:

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  . --no-use-pep517
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ...
...

I believe pypa/setuptools#3291 is another kind of this issue. Note that it neither involves virtualenv, nor Debian/Centos, but sympthoms are exactly the same: system-wide older version of setuptools has higher priority than one that is requested by pypackage.toml.

It’s still worth having the reproducer, when we find out the cause of the main issue here we can check if it addresses the debian case as well.

here’s a hacky patch which fixes the issue by avoiding site in the isolated environment:

$ git show | cat
commit 4c10a5c1bdea62a506ace8e94e87eabc1fd45922
Author: Anthony Sottile <asottile@umich.edu>
Date:   Tue Dec 28 06:53:42 2021 -0800

    hack: workaround broken setuptools distutils pth

diff --git a/src/pip/_internal/build_env.py b/src/pip/_internal/build_env.py
index d326dc8cd..d9f97f52b 100644
--- a/src/pip/_internal/build_env.py
+++ b/src/pip/_internal/build_env.py
@@ -101,21 +101,6 @@ class BuildEnvironment:
                     """
                 import os, site, sys
 
-                # First, drop system-sites related paths.
-                original_sys_path = sys.path[:]
-                known_paths = set()
-                for path in {system_sites!r}:
-                    site.addsitedir(path, known_paths=known_paths)
-                system_paths = set(
-                    os.path.normcase(path)
-                    for path in sys.path[len(original_sys_path):]
-                )
-                original_sys_path = [
-                    path for path in original_sys_path
-                    if os.path.normcase(path) not in system_paths
-                ]
-                sys.path = original_sys_path
-
                 # Second, add lib directories.
                 # ensuring .pth file are processed.
                 for path in {lib_dirs!r}:
diff --git a/src/pip/_vendor/pep517/in_process/_in_process.py b/src/pip/_vendor/pep517/in_process/_in_process.py
index 954a4ab05..c3e5d5bcd 100644
--- a/src/pip/_vendor/pep517/in_process/_in_process.py
+++ b/src/pip/_vendor/pep517/in_process/_in_process.py
@@ -83,6 +83,7 @@ def _build_backend():
         extra_pathitems = backend_path.split(os.pathsep)
         sys.path[:0] = extra_pathitems
 
+    import sitecustomize
     ep = os.environ['PEP517_BUILD_BACKEND']
     mod_path, _, obj_path = ep.partition(':')
     try:
diff --git a/src/pip/_vendor/pep517/wrappers.py b/src/pip/_vendor/pep517/wrappers.py
index e031ed708..223e24b44 100644
--- a/src/pip/_vendor/pep517/wrappers.py
+++ b/src/pip/_vendor/pep517/wrappers.py
@@ -320,7 +320,7 @@ class Pep517HookCaller(object):
             with _in_proc_script_path() as script:
                 python = self.python_executable
                 self._subprocess_runner(
-                    [python, abspath(str(script)), hook_name, td],
+                    [python, '-S', abspath(str(script)), hook_name, td],
                     cwd=self.source_dir,
                     extra_environ=extra_environ
                 )

this appears to “fix” today’s breakage with setuptools 61

I also hit this problem when installing the DonkeyCar system that uses many python machine learning libraries. I downgraded to pip 19.0 and everything seems to be working fine. Thank you Naisy.

if virtualenv inherits the system site package

Presumably when --no-build-isolation is specified? An isolated build ignores the existing installation.

(and that has a setuptoools before 40.8.0)

Well, the point of --no-build-isolation is that the user is agreeing to set up the correct build environment, so that seems like it’s a simple case of user error.

However, in this case @boegel is claiming that he has setuptools 40.8.0 installed, and no other setuptools is present. If that’s the case, and his setuptools doesn’t contain build_meta.__legacy__, then that setuptools installation is somehow broken, because 40.8.0 does contain that backend.

My suspicion is that @boegel is actually mistaken, and there’s another copy of setuptools lingering around somewhere, which is getting picked up and is confusing things. The other possibility is that there’s a bug somewhere - but if that is the case, then we’re currently struggling to reduce the example to something that can easily reproduce/demonstrate that bug.

pip should make sure to never-ever create an isolated build environment that inherits from the system site package

Well, yes, that’s sort of the definition of “isolated” 😃 But I’m not sure how relevant that is in this case, where --no-build-isolation is set.

which is not the case at the moment

That’s what @pganssle is saying in the original post in this thread, yes. But I’m responding to @boegel, who is claiming that he can reproduce the problem without a virtualenv, using --no-build-isolation.

If @boegel’s problem turns out to be real, then I don’t think it’s related to this one (there’s too many differences in the steps to reproduce it). So it should probably be a separate issue. But I’d be interested to understand why he thinks it is the same issue…

Per my Bloomberg colleague @lkollar, it seems that the problem is that pip creates a custom sitecustomize.py file in order to inject itself into the build environment.

Looks to me like it’s just blacklisting the system packages with distutils.get_python_lib, which doesn’t include the system path added by virtualenv. Seems like pep517.envbuild.BuildEnvironment doesn’t have this problem - maybe switch over to using that? Or was there a reason to do it this way?