pip: EnvironmentError: [WinError 5] Access is denied (for some packages)

Environment

• pip version: 19.2.3
• Python version: CPython 3.6.8
• OS: Windows 10
• Git for Windows version: 2.23.0.windows.1

Python is installed in C:\Tools\Python\3.6, and is writable to me as a user. Global packages are often installed there and not with --user. Git for Windows is installed in C:\Users\davisda4\Tools\Git, and many essential tools from Linux will be found there. See the output section for a rendering of directories in the path.

Description

pip install fails to unpack some packages with the typical environment error in the title, but most packages work. Sometimes, rebooting will resolve the issue. The package Django is often affected.

Expected behavior

pip install should work reliably on Windows.

Troubleshooting

Often, I just reboot the system, and it works. This time, it was very persisted so I attempted to analyze and troubleshoot extensively.

• Attempted removing files and directories from %TMP% to clean-up space
• Attempted redefining %TEMP% and %TMP% to be someplace easy to access
• Attempted bypassing cache using --no-cache-dir
• Attempted rebooting the system (finally getting frustrated)
• Attempted upgrading global packages pip, setuptools, wheel, virtualenvwrapper-win, virtualenv, virtualenv-clone
• Attempted running outside of Cmder.net in a pure CMD prompt
• Checked access to %TEMP% directory and available space on the filesystem, reported information is below.

How to Reproduce

The problem is intermittent but has occurred on multiple machines, all running Windows 10. All systems I have tried are within the NLM desktop computing environment, and so there may be security check programs stopping access. This almost always affects Django rather than other packages in my experience.

I believe this to be the symptom of a real bug (whatever the source) that is often masked by the usual permission and space problems. Other users will run into this problem, both in my corporate environment and others. I open this in the spirit of trying to get to the root of the issue.

Output

Output below shows the installation of flake8 working and then Django failing. This is outside the virtualenv, but the same thing happens within a virtualenv, and also with an alternate temporary directory.

C:\Users\davisda4>pip --version
pip 19.2.3 from c:\tools\python\3.6\lib\site-packages\pip (python 3.6)

C:\Users\davisda4>pip install flake8
Looking in indexes: https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/simple
Collecting flake8
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/26/de/3f815a99d86eb10464ea7bd6059c0172c7ca97d4bdcfca41051b388a653b/flake8-3.7.8-py2.py3-none-any.whl (70kB)
     |████████████████████████████████| 71kB 770kB/s
Collecting pyflakes<2.2.0,>=2.1.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/84/f2/ed0ffb887f8138a8fe5a621b8c0bb9598bfb3989e029f6c6a85ee66628ee/pyflakes-2.1.1-py2.py3-none-any.whl (59kB)
     |████████████████████████████████| 61kB 787kB/s
Collecting mccabe<0.7.0,>=0.6.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl
Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting pycodestyle<2.6.0,>=2.5.0 (from flake8)
  Downloading https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/pycodestyle-2.5.0-py2.py3-none-any.whl (51kB)
     |████████████████████████████████| 51kB 825kB/s
Installing collected packages: pyflakes, mccabe, entrypoints, pycodestyle, flake8
Successfully installed entrypoints-0.3 flake8-3.7.8 mccabe-0.6.1 pycodestyle-2.5.0 pyflakes-2.1.1

C:\Users\davisda4>pip install Django
Looking in indexes: https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/simple
Collecting Django
  Using cached https://tools.nlm.nih.gov/artifactory/api/pypi/nlm-pypi-developers/packages/packages/94/9f/a56f7893b1280e5019482260e246ab944d54a9a633a01ed04683d9ce5078/Django-2.2.5-py3-none-any.whl
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\davisda4\\AppData\\Local\\Temp\\pip-unpack-cif5c4t6\\Django-2.2.5-py3-none-any.whl'
Consider using the `--user` option or check the permissions.

Troubleshooting step of checking space and permissions for the pip unpack directory (note that pip was able to create the directory):

C:\Users\davisda4\AppData\Local\Temp\pip-unpack-cif5c4t6>ls

C:\Users\davisda4\AppData\Local\Temp\pip-unpack-cif5c4t6>icacls .
. NT AUTHORITY\SYSTEM:(OI)(CI)(F)
  BUILTIN\Administrators:(OI)(CI)(F)
  NIH\davisda4:(OI)(CI)(F)

Successfully processed 1 files; Failed processing 0 files

C:\Users\davisda4\AppData\Local\Temp\pip-unpack-cif5c4t6>df -kh .
Filesystem                            Size  Used Avail Use% Mounted on
C:/Users/davisda4/AppData/Local/Temp  477G   92G  385G  20% /tmp

Please note that “NIH\davisda4:(OI)(CI)(F)” means that I am the “inherited” owner (OI) and creator (CI) of the directory, and have full access (F).

The system path:

C:\Program Files\VanDyke Software\Clients\
C:\tools\Node\8.15\
C:\Tools\Python\3.6\
C:\tools\Python\3.6\Scripts\
C:\tools\Python\3.5\
C:\tools\Python\3.5\Scripts\
C:\tools\Perl\5.2\Perl64\bin\
C:\tools\Ruby\2.4\bin\
C:\tools\Ruby\2.4\bin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files (x86)\HID Global\ActivClient\
C:\Program Files\HID Global\ActivClient\
C:\Users\nlmoccsswdist.nih\AppData\Local\Microsoft\WindowsApps
C:\Users\nlmoccsswdist.nih\AppData\Roaming\npm
C:\Program Files\dotnet\
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\
C:\Program Files (x86)\Webex\Webex\Applications
c:\Users\nlmoccsswdist.nih\AppData\Roaming\npm
C:\Tools\Java\11\bin\
C:\Tools\Java\11\bin
C:\Tools\Java\11\jre\bin

The user path:

C:\Tools\Python\3.6
C:\Tools\Python\3.6\Scripts
C:\Tools\Perl\5.2\Perl64\bin
C:\Users\davisda4\Tools\apache-ant-1.10.5\bin
C:\Users\davisda4\Tools\apache-maven-3.5.4\bin
C:\Users\davisda4\Tools\SeleniumDrivers
C:\Tools\instantclient_12_2
C:\Users\davisda4\AppData\Roaming\npm
C:\Program Files\Sublime Text 3
C:\Users\davisda4\Tools\xmlsoft\bin
C:\Users\davisda4\Tools\HashiCorp\Vagrant\bin
C:\Users\davisda4\Tools\apache-jena-3.8.0\bat
C:\Users\davisda4\Tools\bin
C:\Users\davisda4\Tools\Git\cmd
C:\Users\davisda4\Tools\Git\mingw64\bin
C:\Users\davisda4\Tools\Git\usr\bin