pip: Upgrading pip fails on Windows when install path is too long

Received the following log on https://github.com/Microsoft/PTVS/issues/782.

Virtual environment is being created at 'C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env'
Virtual environment was successfully created at 'C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env'
----- Installing 'pip' -----
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Using cached pip-7.1.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "C:\Python34\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc' -> 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-h64zdfhc-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\basecommand.py", line 232, in main
          status = self.run(options, args)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\commands\install.py", line 347, in run
          root=options.root_path,
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_set.py", line 543, in install
          requirement.uninstall(auto_confirm=True)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\utils\__init__.py", line 316, in renames
          shutil.move(old, new)
        File "C:\Python34\lib\shutil.py", line 535, in move
          copy2(src, real_dst)
        File "C:\Python34\lib\shutil.py", line 245, in copy2
          copyfile(src, dst, follow_symlinks=follow_symlinks)
        File "C:\Python34\lib\shutil.py", line 109, in copyfile
          with open(dst, 'wb') as fdst:
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-h64zdfhc-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'
----- Failed to install 'pip' -----

The problem seems to be that the entire install path is replicated beneath TEMP, which very quickly exceeds the maximum path length supported by Windows. I guess the aim is to be able to rollback a failed uninstall (which also fails here, and leaves corrupt state), but we may need an alternative to including the full path - maybe generate some sort of map file as well?

About this issue

Original URL
State: closed
Created 9 years ago
Reactions: 3
Comments: 39 (36 by maintainers)

Commits related to this issue

Fixes #3055 Uninstall causes paths to exceed MAX_PATH limit — committed to zooba/pip by zooba 6 years ago
Merge pull request #6029 from zooba/issue-3055 Fixes #3055 Uninstall causes paths to exceed MAX_PATH limit — committed to pypa/pip by pfmoore 6 years ago
Merge #1723 1723: Scheduled weekly dependency update for week 05 r=mythmon a=pyup-bot ### Update [atomicwrites](https://pypi.org/project/atomicwrites) from **1.2.1** to **1.3.0**. *The bot wa... — committed to mozilla/normandy by bors[bot] 5 years ago
Scheduled monthly dependency update for February (#22) ### Update [pip](https://pypi.org/project/pip) from **18.1** to **19.0.1**. <details> <summary>Changelog</summary> #... — committed to vilkasgroup/Pakettikauppa by pyup-bot 5 years ago

Most upvoted comments

I wouldn’t suggest going down the subst or mklink routes at all. pip should just name its temp directory something other than the full path to the eventual/original install location.

The fix in Python 3.6 and Windows 10 is fine, but will never apply to all users.

zooba on Aug 21, 2017

Seems reasonable to me, I’d just say we should use a name that isn’t importable as well.

dstufft on Nov 21, 2018

Thanks @GadgetSteve! I’ll try to summarise what I’ve understood. Here’s how one can workaround this issue as of today:

If you can use Python 3.6 and Windows 10, use them. Along with everything else, it has support for the long paths - since the Anniversary Update of Windows 10.
- You might have to enable long path support on Windows 10 though. [instructions]
If you don’t have the ability to use Windows 10 and Python 3.6, you’ll have to workin the restrictions set by Windows:
- You can use the --cache-dir option to set a cache directory that’s shorter.
- Set TEMP as a shorter path - like C:\Temp
You can try making a long path into a short one with the command line tools subst and mklink commands. (but don’t)

pradyunsg on Sep 21, 2017

Yep, under PEP 517 (now in master, but not yet used for all builds) we set up a build directory (which will still be in $TEMP) and call the backend to build a wheel. We then extract the wheel and move the extracted files. If we extract to a random-named directory in the target location and move, that will skip a whole copy step, which is a definite performance improvement[1]

Legacy non-PEP517 installs go direct via setup.py install, so there’s no wheel or wheel extraction step, so they won’t change (which is fine by me, as you say that code path is destined for removal anyway)

[1] We might get a double improvement - I don’t know much about how anti-virus software works, but if it can recognise that a move doesn’t require a new scan, that could mean that the proposed approach will remove an extra unneeded virus scan as well.

pfmoore on Nov 21, 2018

FWIW, Windows 10 allows the option to disable MAX_PATH limits for “self-certified” applications, which will include Python 3.6. Currently there’s a machine policy that needs to be enabled, and the Python 3.6 installer will prompt users to do that (if they can - need to be admin).

So the issue will go away in the future, and anything else we do now is mitigation for people on existing setups (which obviously has a lot of value, since it’s going to be a long time before everyone is on Python 3.6 on Windows 10).

zooba on Oct 27, 2016