pipenv: pipenv (patched pip) corrupts path for editable requirement on Windows

Issue description

I try to install a package into pipenv’s virtual environment as editable (either using relative path pipenv install -e . or absolute path pipenv install -e D:\long\windows\path\project). I get the error with a traceback with this exception: pipenv.patched.pip._internal.exceptions.InstallationError: D:longwindowspathproject is not a valid editable requirement. It should either be a path to a local project or a VCS URL.... (Full traceback provided below.)

With some adhoc debugging I found that the path is first resolved if it’s relative, and then the absolute path gets corrupted (maybe not only) here (link):

options = shlex.split(options_str)

While running install -e ... commands, this line executes twice:

  1. options_str == '-i https://pypi.org/simple C:\\Users\\user\\AppData\\Local\\pipenv\\pipenv\\Cache', options == ['-i', 'https://pypi.org/simple', 'C:UsersuserAppDataLocalpipenvpipenvCache']

  2. options_str == '-e D:\\long\\windows\\path\\project' options == ['-e', 'D:longwindowspathproject']

shlex.split for some reason removes every backslash from a Windows path, I have tested it in IDLE. I have also tried to run pipenv install -e D:/long/windows/path/project, no changes happened.

When I tried to replicate it with just virtualenv and pip, this line in pip was never executed. (theories ahead) I suppose this is because pipenv creates a temporary requirements.txt to supply it to pip, and somehow it goes wrong. Maybe for local files it should use URLs with file scheme instead of plain absolute paths? ( hinted link)

Steps to replicate

  1. be on Windows (10)

  2. have a Python project which already has associated virtual environment managed by pipenv (I created it with python -m pipenv --python=<python-path> at project root)

3.1. from the root directory of the project, run python -m pipenv install -e . (as in docs) OR 3.2. run python -m pipenv install -e <project-absolute-path>

Expected result

The package is installed into associated virtual environment as editable.

Actual result

The package is not installed, pipenv exits with an error, producing this:

long traceback
d:\long\windows\path\project>python -m pipenv install -e . --dev --verbose
Loading .env environment variables...
Installing -e ....
Resolving -e ....
Installation Succeeded
Pipfile.lock (c606b7) out of date, updating to (916e2e)...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Traceback (most recent call last):
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 675, in <module>
    main()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 661, in main
    _main(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 645, in _main
    resolve_packages(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 592, in resolve
    return resolve_deps(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 918, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 691, in actually_resolve_deps
    resolver.resolve()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 443, in resolve
    constraints = self.constraints
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 401, in constraints
    possible_constraints_list = self.possible_constraints
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 388, in possible_constraints
    possible_constraints_list = [
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 389, in <listcomp>
    install_req_from_parsed_requirement(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 485, in install_req_from_parsed_requirement
    req = install_req_from_editable(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 240, in install_req_from_editable
    parts = parse_req_from_editable(editable_req)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 209, in parse_req_from_editable
    name, url, extras_override = parse_editable(editable_req)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 126, in parse_editable
    raise InstallationError(
pipenv.patched.pip._internal.exceptions.InstallationError: D:longwindowspathproject is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
Locking Failed!
[    ] Locking...
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
    cli()
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 297, in do_install
    raise e
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 281, in do_install
    do_init(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 648, in do_init
    do_lock(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 859, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
  File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 728, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

$ pipenv --support

Pipenv version: '2023.11.17'

Pipenv location: 'C:\\Users\\user\\AppData\\Roaming\\Python\\Python310\\site-packages\\pipenv'

Python location: 'C:\\Program Files\\Python310\\python.exe'

OS Name: 'nt'

User pip version: '23.3.2'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.11',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.19045',
 'python_full_version': '3.10.11',
 'python_version': '3.10',
 'sys_platform': 'win32'}

System environment variables:

…

  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIPENV_VENV_IN_PROJECT
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv-specific environment variables:

  • PIPENV_VENV_IN_PROJECT: 1

Debug-specific environment variables:

  • PATH: ...;C:\Program Files\Python310\Scripts\;C:\Program Files\Python310\;...;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...;C:\Users\user\AppData\Roaming\Python\Python310\Scripts;...;

Contents of Pipfile (‘D:\long\windows\path\project\Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.11"

Contents of Pipfile.lock (‘D:\long\windows\path\project\Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "..."
        },
        "pipfile-spec": 6,
        "requires": {
            "python_full_version": "3.10.11",
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (6 by maintainers)

Most upvoted comments

Can you check out: https://github.com/pypa/pipenv/pull/6276

$ python -m pipenv install -e .
Loading .env environment variables...
Creating a virtualenv for this project
Pipfile: C:\Users\matte\projects\pipenv-triage\pipenv-issue-6068-demo\Pipfile
Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python313\python.exe3.13.0 to create virtualenv...
[ ===] Creating virtual environment...created virtual environment CPython3.13.0.final.0-64 in 273ms
  creator CPython3Windows(dest=C:\Users\matte\projects\pipenv-triage\pipenv-issue-6068-demo\.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==24.2
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\Users\matte\projects\pipenv-triage\pipenv-issue-6068-demo\.venv
Creating a Pipfile for this project...
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Updated Pipfile.lock (494d5b4f482f0ef471f49afe28f00ec1a2ff75da2ce65060d8cabaeb3da2f100)!
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing -e ....
Installation Succeeded
Running $ pipenv lock then $ pipenv sync.
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (81bc18a60e7083da622ebfd05283c28ee2a3d19afcafe316cb5b884cd82b92f2)!
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (2b92f2)...
All dependencies are now up-to-date!
Installing dependencies from Pipfile.lock (2b92f2)...

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
problem-project = {file = ".", editable = true}

[dev-packages]

[requires]
python_version = "3.13"

matte@matteius-proart MINGW64 ~/projects/pipenv-triage/pipenv-issue-6068-demo (master)
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "81bc18a60e7083da622ebfd05283c28ee2a3d19afcafe316cb5b884cd82b92f2"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.13"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "problem-project": {
            "editable": true,
            "file": "."
        }
    },
    "develop": {}
}
```

I mean, I have the fresh project dir at D:\long\windows\path\project with no pipenv’s virtualenv and no pipfiles. I copy the whole project dir to D:\long\project, either renaming project and package or not.

From within original project path, I run python -m pipenv install -e ..\..\..\project (or python -m pipenv install -e D:\long\project, but it comes to shlex.split as relative path still). It fails because pipenv tries to find something at ......project (comment where this particular error is first mentioned).

And absolute paths on different drives don’t work as well.

I expect that it should work because pipenv --help output and man page do not mention particular limitations on a local path for the -e option, and because it works with pip itself as I just tested.

There is pyproject.toml at root of pipenv-issue-6068-demo, I’m not aware what is wrong with that.

Yes, virtualenv + pip install -e . successfully installs the package. This was the session from within this same repo in cmd.exe:

cmd.exe session
d:\long\windows\path\pipenv-issue-6068-demo>python -m virtualenv testenv
created virtual environment CPython3.10.11.final.0-64 in 7748ms
  creator CPython3Windows(dest=D:\long\windows\path\pipenv-issue-6068-demo\testenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\user\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.3.2, setuptools==69.0.3, wheel==0.42.0
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

d:\long\windows\path\pipenv-issue-6068-demo>.\testenv\Scripts\activate

(testenv) d:\long\windows\path\pipenv-issue-6068-demo>pip install -e .
Obtaining file:///D:/long/windows/path/pipenv-issue-6068-demo
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: problem-project
  Building editable for problem-project (pyproject.toml) ... done
  Created wheel for problem-project: filename=problem_project-0.1.0-0.editable-py3-none-any.whl size=1594 sha256=f76979778a63df8472b46c266f35d72fefe6ea130c3bf8329b955e1a92c2a68f
  Stored in directory: C:\Users\user\AppData\Local\Temp\pip-ephem-wheel-cache-q_w0s3yo\wheels\ab\55\bc\7e8acf252f651d596add8fe1a7bd9aa460d7f1424001943dd5
Successfully built problem-project
Installing collected packages: problem-project
Successfully installed problem-project-0.1.0

(testenv) d:\long\windows\path\pipenv-issue-6068-demo>problem_package
Hello World!

upd: and it is actually editable