pip: Re-think error swallowing for de-vendored packages

Pip version: 9.0.1 (should still apply to pip master)
Python version: 3.6
Operating system: Debian Linux

Description:

Currently, in pip._vendor, if DEBUNDLED = True and both the real and the vendored modules don’t import, the ImportError is silently swallowed. Now consider the case that the requested module does exist (example: requests) but is missing a dependency (idna). That leads to a situation where an import of pip._vendor.requests raises ImportError: cannot import name 'requests', without any clue about the original error. Example backtrace (when creating a virtualenv):

Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /tmp/bleh3/bin/python3 - setuptools pkg_resources pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 31, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/vcs/mercurial.py", line 9, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 40, in <module>
ImportError: cannot import name 'requests'
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    load_entry_point('virtualenv==15.1.0', 'console_scripts', 'virtualenv')()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 992, in create_environment
    download=download,
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 922, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /tmp/bleh3/bin/python3 - setuptools pkg_resources pip wheel failed with error code 1

By explicitly raising the original exception instead of swallowing it, the following error is shown instead:

Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /tmp/bleh3/bin/python3 - setuptools pkg_resources pip wheel:
  Traceback (most recent call last):
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 33, in vendored
ModuleNotFoundError: No module named 'pip._vendor.cachecontrol'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 28, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/exceptions.py", line 6, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 64, in <module>
  File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 36, in vendored
  File "/tmp/bleh3/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
  File "/tmp/bleh3/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
  File "/tmp/bleh3/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 4, in <module>
  File "/tmp/bleh3/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/__init__.py", line 98, in <module>
  File "/tmp/bleh3/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/packages.py", line 7, in <module>
ModuleNotFoundError: No module named 'idna'
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    load_entry_point('virtualenv==15.1.0', 'console_scripts', 'virtualenv')()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 992, in create_environment
    download=download,
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 922, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /tmp/bleh3/bin/python3 - setuptools pkg_resources pip wheel failed with error code 1

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 24 (20 by maintainers)

Commits related to this issue

Merge pull request #8391 from VikramJayanthi17/error-swallow-fix Re-think error swallowing for de-vendored packages #5354 — committed to pypa/pip by pradyunsg 4 years ago

Most upvoted comments

This change seems to make optional dependencies of the devendored library hard dependencies of pip, in devendered installation:

Traceback (most recent call last):
  File "pip_sphinxext.py", line 11, in <module>
    from pip._internal.cli import cmdoptions
  File "/build/python-pip/src/pip-20.2/src/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.progress_bars import BAR_TYPES
  File "/build/python-pip/src/pip-20.2/src/pip/_internal/cli/progress_bars.py", line 7, in <module>
    from pip._vendor import six
  File "/build/python-pip/src/pip-20.2/src/pip/_vendor/__init__.py", line 83, in <module>
    vendored("requests.packages.urllib3.contrib.ntlmpool")
  File "/build/python-pip/src/pip-20.2/src/pip/_vendor/__init__.py", line 33, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "/usr/lib/python3.8/site-packages/urllib3/contrib/ntlmpool.py", line 9, in <module>
    from ntlm import ntlm
ModuleNotFoundError: No module named 'ntlm'

Is this intended?

felixonmars on Aug 12, 2020

@rohitsanj please inform earlier if you already have started working over the issue. It would be inconvenient for two people working on the same issue without knowing.

Although, this was not a major fix, this kind of conflict may arise across other issues. So just let others know that you are already working over the issue.

Thanks. 😃

gutsytechster on Feb 26, 2020

This issue is fixed by https://github.com/pypa/pip/pull/8391 and can be closed.

deveshks on Jun 5, 2020

Aah you are correct @McSinyx . Appreciate the detailed explanation 😃

deveshks on Apr 3, 2020