pipenv: Using 2023.9.7 editable package installation fails with build isolation disabled when depending upon setuptools

Issue description

When performing an editable install of a package that depends upon setuptools with build isolation disabled (see https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-no-build-isolation) pipenv clobbers setuptools while it’s being used to install an editable package. I assume this is because installs happen in parallel and setuptools gets swapped out from under the editable package while it’s being installed.

Expected result

I’m not sure when this last worked, we hadn’t upgraded pipenv for quite a while. But at some point this seemed to work, i.e we could install an editable package with pipenv that depends upon setuptools.

Actual result

Removing .venv/
Removing Pipfile
Removing Pipfile.lock
Creating a virtualenv for this project...
Pipfile: /home/stewart/src/pipenv_editable_install_repro/Pipfile
Using /usr/bin/python (3.8.10) to create virtualenv...
created virtual environment CPython3.8.10.final.0-64 in 49ms
  creator CPython3Posix(dest=/home/stewart/src/pipenv_editable_install_repro/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/stewart/.local/share/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.1.2, wheel==0.41.2
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/stewart/src/pipenv_editable_install_repro/.venv
Creating a Pipfile for this project...
The flag --skip-lock has been reintroduced (but is not recommended).  Without the lock resolver it is difficult to manage multiple package 
indexes, and hash checking is not provided.  However it can help manage installs with current deficiencies in locking across platforms.
Installing -e ./requires-setuptools...
Resolving -e ./requires-setuptools...
Added requires-setuptools to Pipfile's [packages] ...
✔ Installation Succeeded
Installing dependencies from Pipfile...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (c2fec5ff1a495cbfa1fcbdd572d8f990a3559d4d8b69565aa35d34409ad990a3)!
Installing dependencies from Pipfile.lock (d990a3)...
[pipenv.exceptions.InstallError]: Obtaining file:///home/stewart/src/pipenv_editable_install_repro/requires-setuptools (from -r /tmp/pipenv-mql58ofv-requirements/pipenv-anmjg6eu-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Checking if build backend supports build_editable: started
[pipenv.exceptions.InstallError]:   Checking if build backend supports build_editable: finished with status 'done'
[pipenv.exceptions.InstallError]:   Preparing editable metadata (pyproject.toml): started
[pipenv.exceptions.InstallError]:   Preparing editable metadata (pyproject.toml): finished with status 'done'
[pipenv.exceptions.InstallError]: ERROR: Exception:
[pipenv.exceptions.InstallError]: Traceback (most recent call last):
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
[pipenv.exceptions.InstallError]:     status = run_func(*args)
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/cli/req_command.py", line 248, in wrapper
[pipenv.exceptions.InstallError]:     return func(self, options, args)
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/commands/install.py", line 377, in run
[pipenv.exceptions.InstallError]:     requirement_set = resolver.resolve(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
[pipenv.exceptions.InstallError]:     collected = self.factory.collect_root_requirements(root_reqs)
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
[pipenv.exceptions.InstallError]:     req = self._make_requirement_from_install_req(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py", line 453, in _make_requirement_from_install_req
[pipenv.exceptions.InstallError]:     cand = self._make_candidate_from_link(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py", line 185, in _make_candidate_from_link
[pipenv.exceptions.InstallError]:     self._editable_candidate_cache[link] = EditableCandidate(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 322, in __init__
[pipenv.exceptions.InstallError]:     super().__init__(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 157, in __init__
[pipenv.exceptions.InstallError]:     self.dist = self._prepare()
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 226, in _prepare
[pipenv.exceptions.InstallError]:     dist = self._prepare_distribution()
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 332, in _prepare_distribution
[pipenv.exceptions.InstallError]:     return self._factory.preparer.prepare_editable_requirement(self._ireq)
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 709, in prepare_editable_requirement
[pipenv.exceptions.InstallError]:     dist = _get_prepared_distribution(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
[pipenv.exceptions.InstallError]:     abstract_dist.prepare_distribution_metadata(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/distributions/sdist.py", line 61, in prepare_distribution_metadata
[pipenv.exceptions.InstallError]:     self.req.prepare_metadata()
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/req/req_install.py", line 557, in prepare_metadata
[pipenv.exceptions.InstallError]:     self.metadata_directory = generate_editable_metadata(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/operations/build/metadata_editable.py", line 37, in generate_editable_metadata
[pipenv.exceptions.InstallError]:     distinfo_dir = backend.prepare_metadata_for_build_editable(metadata_dir)
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_internal/utils/misc.py", line 731, in prepare_metadata_for_build_editable
[pipenv.exceptions.InstallError]:     return super().prepare_metadata_for_build_editable(
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_impl.py", line 246, in prepare_metadata_for_build_editable
[pipenv.exceptions.InstallError]:     return self._call_hook('prepare_metadata_for_build_editable', {
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
[pipenv.exceptions.InstallError]:     raise BackendUnavailable(data.get('traceback', ''))
[pipenv.exceptions.InstallError]: pipenv.patched.pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
[pipenv.exceptions.InstallError]:   File "/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
[pipenv.exceptions.InstallError]:     obj = import_module(mod_path)
[pipenv.exceptions.InstallError]:   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
[pipenv.exceptions.InstallError]:     return _bootstrap._gcd_import(name[level:], package, level)
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
[pipenv.exceptions.InstallError]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[pipenv.exceptions.InstallError]:   File "/home/stewart/src/pipenv_editable_install_repro/.venv/lib/python3.8/site-packages/setuptools/__init__.py", line 15, in <module>
[pipenv.exceptions.InstallError]:   File "/home/stewart/src/pipenv_editable_install_repro/.venv/lib/python3.8/site-packages/setuptools/version.py", line 1, in <module>
[pipenv.exceptions.InstallError]:   File "/home/stewart/src/pipenv_editable_install_repro/.venv/lib/python3.8/site-packages/setuptools/_importlib.py", line 41, in <module>
[pipenv.exceptions.InstallError]:   File "/home/stewart/src/pipenv_editable_install_repro/.venv/lib/python3.8/site-packages/setuptools/extern/__init__.py", line 52, in create_module
[pipenv.exceptions.InstallError]:   File "/home/stewart/src/pipenv_editable_install_repro/.venv/lib/python3.8/site-packages/setuptools/extern/__init__.py", line 44, in load_module
[pipenv.exceptions.InstallError]: ImportError: The 'importlib_metadata' package is required; normally this is bundled with this package so if you get this warning, consult the packager of your distribution.
ERROR: Couldn't install package: {}
 Package installation failed...

Steps to replicate

Clone https://github.com/stewartmiles/pipenv_editable_install_repro
Run requires-setuptools/install.sh on a Linux box / macOS box or you can run the equivalent commands from your Windows terminal of choice.

$ pipenv --support

Pipenv version: '2023.9.7'

Pipenv location: '/home/stewart/.local/pipx/venvs/pipenv/lib/python3.8/site-packages/pipenv'

Python location: '/home/stewart/.local/pipx/venvs/pipenv/bin/python'

OS Name: 'posix'

User pip version: '23.2.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.0-69-generic',
 'platform_system': 'Linux',
 'platform_version': '#76~20.04.1-Ubuntu SMP Mon Mar 20 15:54:19 UTC 2023',
 'python_full_version': '3.8.10',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

SHELL
PIPENV_VENV_IN_PROJECT
TERMCAP
SSH_AUTH_SOCK
WINDOW
SSH_AGENT_PID
PWD
LOGNAME
XDG_SESSION_TYPE
MOTD_SHOWN
HOME
LANG
LS_COLORS
SSH_CONNECTION
LESSCLOSE
XDG_SESSION_CLASS
TERM
LESSOPEN
USER
XLA_FLAGS
SHLVL
XDG_SESSION_ID
KUBECONFIG
LD_LIBRARY_PATH
XDG_RUNTIME_DIR
SSH_CLIENT
XDG_DATA_DIRS
PATH
STY
DBUS_SESSION_BUS_ADDRESS
SSH_TTY
USE_GKE_GCLOUD_AUTH_PLUGIN
OLDPWD
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_VENV_IN_PROJECT: true

Debug–specific environment variables:

PATH: /usr/local/cuda/bin:/usr/local/nvidia/bin:/home/stewart/bin/gradle/bin:/home/stewart/bin/cmake/bin:/usr/local/cuda/bin:~/.local/bin:/home/stewart/.local/bin:/home/stewart/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/stewart/bin/node/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/stewart/src/pipenv_editable_install_repro

Contents of Pipfile (‘/home/stewart/src/pipenv_editable_install_repro/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requires-setuptools = {file = "requires-setuptools", editable = true}

[dev-packages]

[requires]
python_version = "3.8"
python_full_version = "3.8.10"

Contents of Pipfile.lock (‘/home/stewart/src/pipenv_editable_install_repro/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "c2fec5ff1a495cbfa1fcbdd572d8f990a3559d4d8b69565aa35d34409ad990a3"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_full_version": "3.8.10",
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "requires-setuptools": {
            "editable": true,
            "file": "requires-setuptools"
        },
        "setuptools": {
            "hashes": [
                "sha256:00478ca80aeebeecb2f288d3206b0de568df5cd2b8fada1209843cc9a8d88a48",
                "sha256:af3d5949030c3f493f550876b2fd1dd5ec66689c4ee5d5344f009746f71fd5a8"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==68.2.0"
        }
    },
    "develop": {}
}

About this issue

Original URL
State: open
Created 10 months ago
Comments: 20 (13 by maintainers)

Most upvoted comments

@stewartmiles might be worth trying todays release just in case there was an issue resolving the specific build requirement/dependency, since ignore_compatibility was defaulted to True in yesterdays release. That being said …

It sounds like supporting build time dependencies of editable wheels isn’t a thing pipenv would like to do then?

I guess I don’t have a solid answer on this, my initial take was we are supporting it via pip’s level of support. The requirementslib code that maybe did that was also maybe not being invoked, because there was a lot of cruft code from the time that pipenv’s resolver lived in requirementslib. Maybe one day I can pick apart more if it was truly doing something helpful with regards to build time dependencies, but ideally I would need to know what the last version of pipenv it did work properly on – its possible we could be looking up the wrong tree.

As you describe, use categories and basically install setuptools and build dependencies manually first followed by packages that require setuptools at build time.

Pin setuptools to the version bundled with pipenv and manually update this on each pipenv upgrade.

If point two solves it, then are there really other build dependencies to put in a separate package group besides setuptools (pt 1). I think the advantage of having a separate group, is that while named requirements get installed before editable, you would guarantee setuptools is the same for even named requirements ahead of invoking the default group install.

Ditch using packages that need to be built (e.g editable packages) and instead build them using pip wheel and maintain a private repository. This is seriously sucky from a developer workflow point of view IMHO.

Understood.

Bring back pipenv sync --sequential, it would be slow but it would avoid this issue, what do you think?

I am not convinced on this simply because there was never a way to specify the ordering of the sequential install, so I feel its a can of worms to try and support it.

I’ll look more at your reproduction example when I have more time and see if there is another angle to consider.

matteius on Sep 8, 2023