poetry: ValueError, whl in cache does not exist when running poetry install again

Also have this issue with 1.1.8 on Windows 10. My workaround is to go to the path specified, delete the cache and run poetry install again.

e.g.

PS C:\GithubTech\ngx\ngx-cli> poetry install
Installing dependencies from lock file

Package operations: 32 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)

  ValueError

  File \C:\Users\julie\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl does not exist

  at c:\users\julie\.pyenv\pyenv-win\versions\3.9.6\lib\site-packages\poetry\core\packages\file_dependency.py:40 in __init__
       36│             except FileNotFoundError:
       37│                 raise ValueError("Directory {} does not exist".format(self._path))
       38│
       39│         if not self._full_path.exists():
    →  40│             raise ValueError("File {} does not exist".format(self._path))
       41│
       42│         if self._full_path.is_dir():
       43│             raise ValueError("{} is a directory, expected a file".format(self._path))
       44│

Go to C:\Users\julie\AppData\Local\pypoetry\Cache, delete everything.

Run poetry install again and:

PS C:\GithubTech\ngx\ngx-cli> poetry install
Installing dependencies from lock file

Package operations: 32 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)
  • Installing atomicwrites (1.4.0)
  • Installing attrs (21.2.0)
  • Installing backports.entry-points-selectable (1.1.0)
  • Installing filelock (3.0.12)
  • Installing iniconfig (1.1.1)
  • Installing lazy-object-proxy (1.6.0)
  • Installing packaging (21.0)
  • Installing platformdirs (2.3.0)
  • Installing pluggy (0.13.1)
  • Installing py (1.10.0)
  • Installing six (1.16.0)
  • Installing toml (0.10.2)
  • Installing wrapt (1.12.1)
  • Installing astroid (2.7.3)
  • Installing identify (2.2.14)
  • Installing mccabe (0.6.1)
  • Installing mypy-extensions (0.4.3)
  • Installing pytest (6.2.4)
  • Installing typing-extensions (3.10.0.2)
  • Installing cfgv (3.3.1)
  • Installing nodeenv (1.6.0)
  • Installing isort (5.9.3)
  • Installing pyyaml (5.4.1)
  • Installing virtualenv (20.7.2)
  • Installing mypy (0.910)
  • Installing pylint (2.10.2)
  • Installing pre-commit (2.14.0)
  • Installing yapf (0.31.0)
  • Installing pytest-mock (3.6.1)

Installing the current project: ngx-cli (0.1.0)

Hope it helps other people in the meantime!

I found some much better PRs that already had some eyes, not sure what the path to get the PRs merged is.

1.1: #4549 1.2+: #4531

I found the culprit!

https://github.com/python-poetry/poetry/blob/e180aab14bdc1d0c9e76b2717fd0de189d26fddd/poetry/installation/executor.py#L673-L680

At this point, archive is always going to be a file URL because it’s been downloaded and cached. But Link.path is broken for file URLs on Windows, so the patch is as follows:

https://github.com/python-poetry/poetry-core/blob/ad33bc2f92be03dc5b31a666664903c439fb1173/poetry/core/packages/utils/link.py#L9 becomes:

from .utils import path_to_url, url_to_path

and

https://github.com/python-poetry/poetry-core/blob/ad33bc2f92be03dc5b31a666664903c439fb1173/poetry/core/packages/utils/link.py#L103-L105 becomes

    @property
    def path(self) -> str:
        if self.url.startswith("file:"):
            p = str(url_to_path(self.url))
        else:
            p = urlparse.unquote(urlparse.urlsplit(self.url)[2])
        return p

Unfortunately this impacts a number of places where improper path handling happens, mainly in the executor module. It’s a rabbit hole for sure and I need to look at my local edits and compare with a fresh clone to get ALL of the spots I changed code.

In hindsight, it may be better to leave Link.path alone and instead fix all of the misuses of it.

After patching Link.path to properly handle Windows file URIs, poetry still fails to install the cached package:

  Stack trace:

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1296 in _run
      1294│                 return subprocess.call(cmd, stderr=subprocess.STDOUT, env=env, **kwargs)
      1295│             else:
    → 1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )

  1  ~\AppData\Local\Programs\Python\Python39\lib\subprocess.py:424 in check_output
       422│         kwargs['input'] = empty
       423│
    →  424│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       425│                **kwargs).stdout
       426│

  CalledProcessError

  Command 'C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9\Scripts\python.exe C:\Users\user\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-21.0.1-py3-none-any.whl\pip install --disable-pip-version-check --prefix C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9 --no-deps file:\C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl' returned non-zero exit status 1.

  at ~\AppData\Local\Programs\Python\Python39\lib\subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│
       532│

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\pip.py:47 in pip_install
       45│
       46│     try:
    →  47│         return environment.run_pip(*args)
       48│     except EnvCommandError as e:
       49│         if sys.version_info < (3, 7) and not is_wheel:
                                                                                                           
  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1264 in run_pip
      1262│         pip = self.get_pip_command(embedded=True)
      1263│         cmd = pip + list(args)
    → 1264│         return self._run(cmd, **kwargs)
      1265│
      1266│     def run_python_script(self, content: str, **kwargs: Any) -> str:
                                                                                                           
  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1547 in _run
      1545│     def _run(self, cmd: List[str], **kwargs: Any) -> Optional[int]:
      1546│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    → 1547│         return super()._run(cmd, **kwargs)
      1548│
      1549│     def get_temp_environ(

  EnvCommandError

  Command C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9\Scripts\python.exe C:\Users\user\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-21.0.1-py3-none-any.whl\pip install --disable-pip-version-check --prefix C:\Users\user\AppData\Local\pypoetry\Cache\virtualenvs\poetry-cache-test-keI1PhHb-py3.9 --no-deps file:\C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing-2.4.7-py2.py3-none-any.whl errored with the following return code 1, and output:
  ERROR: Invalid requirement: '\\C:\\Users\\user\\AppData\\Local\\pypoetry\\Cache\\artifacts\\92\\0f\\cf\\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\\pyparsing==2.4.7'
  Hint: It looks like a path. File '\C:\Users\user\AppData\Local\pypoetry\Cache\artifacts\92\0f\cf\effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3\pyparsing==2.4.7' does not exist.


  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1300 in _run
      1296│                 output = subprocess.check_output(
      1297│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1298│                 )
      1299│         except CalledProcessError as e:
    → 1300│             raise EnvCommandError(e, input=input_)
      1301│
      1302│         return decode(output)
      1303│
      1304│     def execute(self, bin: str, *args: str, **kwargs: Any) -> Optional[int]:

The following error occurred when trying to handle this error:


  Stack trace:

  5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:246 in _execute_operation
      244│
      245│             try:
    → 246│                 result = self._do_execute_operation(operation)
      247│             except EnvCommandError as e:
      248│                 if e.e.returncode == -2:

  4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:323 in _do_execute_operation
      321│             return 0
      322│
    → 323│         result = getattr(self, f"_execute_{method}")(operation)
      324│
      325│         if result != 0:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:464 in _execute_install
      462│
      463│     def _execute_install(self, operation: Union[Install, Update]) -> int:
    → 464│         status_code = self._install(operation)
      465│
      466│         self._save_url_reference(operation)

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:509 in _install
      507│         )
      508│         self._write(operation, message)
    → 509│         return self.pip_install(str(archive), upgrade=operation.job_type == "update")
      510│
      511│     def _update(self, operation: Union[Install, Update]) -> int:

  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\installation\executor.py:131 in pip_install
      129│         # req = Link(req).path if isinstance(req, str) else req
      130│         try:
    → 131│             func(req, self._env, upgrade=upgrade)
      132│         except EnvCommandError as e:
      133│             output = decode(e.e.output)

  PoetryException

  Failed to install file:/C:/Users/user/AppData/Local/pypoetry/Cache/artifacts/92/0f/cf/effdcd5d76a6186df0969f85b3b030284ff8058936d5016540b5258ea3/pyparsing-2.4.7-py2.py3-none-any.whl

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\pip.py:60 in pip_install
       56│                     *env.get_pip_command(),
       57│                     *args,
       58│                     env={**os.environ, "PYTHONPATH": str(env.purelib)},
       59│                 )
    →  60│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       61│
       62│
       63│ def pip_editable_install(directory: Path, environment: Env) -> Union[int, str]:
       64│     return pip_install(

This is because Executor.pip_install is being called with a string req containing - yet again - a Windows file URI. When this URI string gets passed to pip_install, it’s used to construct a Path, which, like the original Link.path, also doesn’t quite understand URIs:

>>> path = 'file:///C:/Users/foo/bar'
>>> Path(path)
WindowsPath('file:/C:/Users/foo/bar')

We can get around this by using the fixed Link.path to “sanitize” string inputs to Executor.pip_install, stripping away the URI prefix before passing it to the utility pip_install function.

# Sanitize req of type str
req = Link(req).path if isinstance(req, str) else req
try:
    func(req, self._env, upgrade=upgrade)

With these two changes, I can successfully install from the cache.

I ran into this same issue today.

Calling urllib.parse.urlsplit on a Windows file scheme URI (file:///C:/Users/...) leaves the extra leading / in the result’s path attribute.

We can strip this character by passing the malformed path into urllib.request.url2pathname() to get a valid Windows-style path string.

>>> url = 'file:///C:/Users/foo/bar'
>>> split = urllib.parse.urlsplit(url)
>>> split.path
'/C:/Users/foo/bar'
>>> urllib.request.url2pathname(split.path)
'C:\\Users\\foo\\bar'

Note that url2pathname is only recommended for use on file scheme URIs, and it basically just wraps urllib.parse.unquote on non-Windows systems.[0] On Windows, it ends up using nturl2path.url2pathname.[1]

In fact, pip also uses url2pathname, along with some extra logic to handle the special case of a Windows UNC path (e.g. for shared network folders).[2]

[0] https://github.com/python/cpython/blob/3.9/Lib/urllib/request.py#L1675 [1] https://github.com/python/cpython/blob/v3.6.1/Lib/nturl2path.py#L3 [2] https://github.com/pypa/pip/blob/9.0.1/pip/download.py#L442

I made two PRs:

https://github.com/python-poetry/poetry/pull/4651 and https://github.com/python-poetry/poetry-core/pull/222

I made them with the suggestions from this issue. I can not repro @vsutardja’s issue that required a fixed executor (when I applied the fixes to Link.path my repro went away). I don’t know if the fixes @jgentil is mentioning in executor.py still need to be put into the PRs.

using version 1.1.10 here, same issue downgraded to 1.0.8 works fine