pre-commit: [2.2.0] `pre-commit install --install-hooks` fails with: ModuleNotFoundError: No module named 'pip._vendor.six'

Hi!

I’m having trouble to set activate an existing .pre-commit-config.yaml on Debian bullseye. pre-commit crashes saying ModuleNotFoundError: No module named 'pip._vendor.six'. Any idea why?

Here’s what I did on the shell for reproduction:

# cd "$(mktemp -d)"

# git clone https://github.com/hartwork/jawanndenn
Cloning into 'jawanndenn'...
remote: Enumerating objects: 1307, done.
remote: Total 1307 (delta 0), reused 0 (delta 0), pack-reused 1307
Receiving objects: 100% (1307/1307), 223.04 KiB | 951.00 KiB/s, done.
Resolving deltas: 100% (876/876), done.

# cd jawanndenn/

# virtualenv --python=python3.8 py38
created virtual environment CPython3.8.2.final.0-64 in 141ms
  creator CPython3Posix(dest=/tmp/tmp.5152OOPQq6/py38, clear=False, global=False)
  seeder FromAppData(download=False, setuptools=latest, msgpack=latest, pytoml=latest, lockfile=latest, requests=latest, packaging=latest, appdirs=latest, six=latest, retrying=latest, progress=latest, chardet=latest, html5lib=latest, webencodings=latest, colorama=latest, wheel=latest, distlib=latest, certifi=latest, idna=latest, pep517=latest, urllib3=latest, contextlib2=latest, pkg_resources=latest, distro=latest, pip=latest, pyparsing=latest, CacheControl=latest, via=copy, app_data_dir=/home/user/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

# source py38/bin/activate

# pip install pre-commit
Collecting pre-commit
  Using cached pre_commit-2.2.0-py2.py3-none-any.whl (170 kB)
Collecting cfgv>=2.0.0
  Using cached cfgv-3.1.0-py2.py3-none-any.whl (7.3 kB)
Collecting nodeenv>=0.11.1
  Using cached nodeenv-1.3.5-py2.py3-none-any.whl (21 kB)
Collecting toml
  Using cached toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting virtualenv>=15.2
  Using cached virtualenv-20.0.15-py2.py3-none-any.whl (4.6 MB)
Processing /home/user/.cache/pip/wheels/13/90/db/290ab3a34f2ef0b5a0f89235dc2d40fea83e77de84ed2dc05c/PyYAML-5.3.1-cp38-cp38-linux_x86_64.whl
Collecting identify>=1.0.0
  Using cached identify-1.4.13-py2.py3-none-any.whl (97 kB)
Requirement already satisfied: appdirs<2,>=1.4.3 in ./py38/lib/python3.8/site-packages (from virtualenv>=15.2->pre-commit) (1.4.3)
Requirement already satisfied: distlib<1,>=0.3.0 in ./py38/lib/python3.8/site-packages (from virtualenv>=15.2->pre-commit) (0.3.0)
Collecting filelock<4,>=3.0.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Requirement already satisfied: six<2,>=1.9.0 in ./py38/lib/python3.8/site-packages (from virtualenv>=15.2->pre-commit) (1.14.0)
Installing collected packages: cfgv, nodeenv, toml, filelock, virtualenv, pyyaml, identify, pre-commit
Successfully installed cfgv-3.1.0 filelock-3.0.12 identify-1.4.13 nodeenv-1.3.5 pre-commit-2.2.0 pyyaml-5.3.1 toml-0.10.0 virtualenv-20.0.15

# pre-commit install --install-hooks
pre-commit installed at .git/hooks/pre-commit
[INFO] Initializing environment for https://github.com/asottile/pyupgrade.
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-isort.
[INFO] Initializing environment for https://github.com/Lucas-C/pre-commit-hooks-markup.
[INFO] Installing environment for https://github.com/asottile/pyupgrade.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/bin/python', '/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/bin/pip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    Traceback (most recent call last):
      File "/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/bin/pip", line 5, in <module>
        from pip._internal.cli.main import main
      File "/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 10, in <module>
        from pip._internal.cli.autocompletion import autocomplete
      File "/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/lib/python3.8/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
        from pip._internal.cli.main_parser import create_main_parser
      File "/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/lib/python3.8/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
        from pip._internal.cli import cmdoptions
      File "/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/lib/python3.8/site-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
        from pip._internal.exceptions import CommandError
      File "/home/user/.cache/pre-commit/repoudqv1j_3/py_env-python3.8/lib/python3.8/site-packages/pip/_internal/exceptions.py", line 10, in <module>
        from pip._vendor.six import iteritems
    ModuleNotFoundError: No module named 'pip._vendor.six'

Check the log at /home/user/.cache/pre-commit/pre-commit.log

The log file is here: pre-commit.log

Any ideas what I can do to get things to work?

Thanks and best, Sebastian

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 30 (11 by maintainers)

Commits related to this issue

Most upvoted comments

This is basically a result of Debian’s policy of debundling software that has code vendored in it, and doing a not-great job of it. pip has a strong rationale to not be debundled, as documented in our “Vendoring Policy”.

As far as I know, Debian is the only “major” Linux distribution that does major patches to both Python and pip, and these patches often cause breakages and bugs like these. There are legitimate reasons to avoid vendoring. That said, multiple other distributions who have similar policies do take the effort to make sure pip/Python isn’t broken by bringing the versions they distribute as close to the upstream’s distribution as possible, despite the fact that doing so introduces complexity into their everything-built-from-source pipeline.

Filing a bug with Debian is the best/only reasonable recourse here since no one else “upstream” of them can make changes to actually fix the issues created by their patches. They actively do not heed to pressure/pursuation from upstream pip, even when it causes significant breakage for a lot of their users (like in pypa/pip#5599).

And, since Canonical and various other organisations provide LTS support for Debian-based distros, even if they do change course and fix these issues, we’d continue to see user reports since a significant portion of users would still be using these severely patched pip/Python for years.

Basically, this is a Debian bug, one that they have refused to fix as a matter of policy for years while layering on more-and-more hacks to make it work, and there’s nothing that we can do about it.

@hartwork (virtualenv 20 author/maintainer here) The invariant that’s broken with Debian repackaging is that virtualenv expects wheels provided to work by just extracting them into a virtual environments site-package (aka where there are no system packages available). By stripping out the vendored packages I’m not sure how Debian expects the wheel to still work 🤔 (given that it will not be installed at system level).

CC @gaborbernat since you’re probably going to see this same trace in the pypa/virtualenv tracker at some point

“vendoring is bad” but taken out of context and to an extreme level 😆 (the unvendor doesn’t actually do anything in a virtualenv! it’s not a problem of system packages there) – can’t wait for the many pip bug reports when someone upgrades/downgrades one of those packages and breaks pip – pypa is going to have a hell of a ride 😃

I managed to figure out debian’s reporting, took a while: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956144

I think unbundling is ok for binaries and compiled languages. For dynamic languages like Python bundle-ing is the way you guarantee your dependencies are guaranteed to be satisfied.