pip: Upgrading to pip 10: It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

  • Pip version: 10.0.0
  • Python version: 2.7
  • Operating system: Amazon ECS-Optimized Amazon Linux AMI 2017.09.i

Description:

I am trying to install docker-py, it is a usual part of our infrastructure setup workflow and we run it for quite some time. I get the following error for some packages:

Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Everything works as expected with pip 9.0.2. Did something change? What can I do to fix this? (except pinning the version of pip to 9.0.2 or 9.0.3)

What I’ve run:

First installation attempt with pip 10

# /usr/local/bin/pip2 install docker-py
Collecting docker-py
  Using cached docker_py-1.10.6-py2.py3-none-any.whl
Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python2.7/site-packages (from docker-py) (1.11.0)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/local/lib/python2.7/site-packages (from docker-py) (3.5.0.1)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/site-packages (from docker-py) (1.0.22)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python2.7/site-packages (from docker-py) (0.47.0)
Collecting requests!=2.11.0,>=2.5.2 (from docker-py)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting docker-pycreds>=0.2.1 (from docker-py)
  Using cached docker_pycreds-0.2.2-py2.py3-none-any.whl
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py) (2018.1.18)
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.11.0,>=2.5.2->docker-py)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Installing collected packages: chardet, requests, docker-pycreds, docker-py
  Found existing installation: chardet 2.0.1
Cannot uninstall 'chardet'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Downgraded to pip 9.0.3 and then got this:

# /usr/local/bin/pip2 install docker-py
Collecting docker-py
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading docker_py-1.10.6-py2.py3-none-any.whl (50kB)
    100% |████████████████████████████████| 51kB 4.8MB/s
Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python2.7/site-packages (from docker-py)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /usr/local/lib/python2.7/site-packages (from docker-py)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /usr/local/lib/python2.7/site-packages (from docker-py)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python2.7/site-packages (from docker-py)
Collecting requests!=2.11.0,>=2.5.2 (from docker-py)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 8.2MB/s
Collecting docker-pycreds>=0.2.1 (from docker-py)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading docker_pycreds-0.2.2-py2.py3-none-any.whl
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/site-packages (from requests!=2.11.0,>=2.5.2->docker-py)
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.11.0,>=2.5.2->docker-py)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 7.7MB/s
Installing collected packages: chardet, requests, docker-pycreds, docker-py
  Found existing installation: chardet 2.0.1
    DEPRECATION: Uninstalling a distutils installed project (chardet) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling chardet-2.0.1:
      Successfully uninstalled chardet-2.0.1
  Found existing installation: requests 1.2.3
    Uninstalling requests-1.2.3:
      Successfully uninstalled requests-1.2.3
Successfully installed chardet-3.0.4 docker-py-1.10.6 docker-pycreds-0.2.2 requests-2.18.4
You are using pip version 9.0.3, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I also noticed issues with installing awscli, it complains some packages are not installed. (Installing them first fixes the issue though).

If i try to forcereinstall awscli for example, I get the same error for PyYAML:

 It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. ```

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 14
  • Comments: 41 (10 by maintainers)

Commits related to this issue

Most upvoted comments

I used follow steps and resolved this problem

  1. Reduced version: pip install --upgrade --force-reinstall pip==9.0.3
  2. Tried to re-install package: pip install xxx --disable-pip-version-check
  3. At last, recover the latest version for pip: pip install --upgrade pip
+207
wangxf1987 on Dec 20, 2018

This seems to fix the issue for me: https://github.com/blockstack/blockstack-core/issues/504

In the orignal case above, that’d be:

pip install docker-py --ignore-installed PyYAML
+118
JohnBDonner on Aug 23, 2018

Se #4805 and #3389 for the history. Basically, pip cannot properly uninstall packages installed by “pure” distutils, because distutils doesn’t record enough metadata for us to do so. We’ve previously removed the installation metadata, so that it looks like we did the install, but we’ve had to leave files behind. That causes issues. Since pip 8, we’ve been trying to stop doing this, because it’s a cause of problems, but our initial attempt was reverted because it affected too many people at the time. With pip 10, we’ve finally stopped trying to uninstall distutils packages, and now we report the problem to the user as you see here.

Basically, if you (or some infrastructure you use) installed a package using distutils, you need to manage (and in particular) uninstall it “using distutils”. Unfortunately distutils doesn’t include an uninstall command, so “uninstall using distutils” means manually removing the package.

+97
pfmoore on Apr 16, 2018

Try removing package manually from “site-packages”. This works perfect!

+45
shubhamyedage on May 9, 2018

This is insane. Installations are being broken because of some religion-like believing. if some setup.py entry of some package touches urllib3, then there is no way to tells it to ignore it. In our case, we must upgrade urllib3 because in Ubuntu 14 the installed version of pip (v1.5.x) refuses to install packages from https URLs.

The point is that nobody would upgrade “system” packages if they would not stop working. In linux kernel project this behavior is “breaking userspace” and Torvalds does not react nicely to it. It is incredible python people takes it so naturally fine.

At least you should warn people in pip messages with something like “this version of pip will refuse to work if it means to upgrade system packages, do this and do that. Use pip version 9.x if you are working with old installers or old systems.” or similar.

+11
pabloa on Sep 21, 2018

Don’t know who you are, but I love you.

+8
AleJuliet on Sep 18, 2018

The solution is to uninstall it manually, so go to…/anaconda3/lib/python3.*/site-packages/ and delete all files and folders of the package

+3
swishderzy on Jan 30, 2019

It wasn’t intended to solve the problem. It was intended to explain why things are the way they are, and why pip can’t solve the problem. Just because people are sad about the situation doesn’t alter it.

+3
pfmoore on Nov 22, 2018

@ramonamis sudo pip install --ignore-installed PyYAML

This successfully upgraded it for me.

+2
andrewcar on Feb 13, 2019

Remove Dist Package and RUN

sudo rm -rf /usr/lib/python3/dist-packages/yaml

sudo rm -rf /usr/lib/python3/dist-packages/PyYAML-*

Removing folder from distutils works

+2
jinnabaalu on Dec 26, 2018

I agree with @pabloa - This has been going on forever now, a vanilla install of CentOS 7 cannot install docker-compose because of this.

And yes, @JohnBDonner, that little flag just saved me a LOT of time. THANK YOU

+2
AddoSolutions on Oct 3, 2018

I used follow steps and resolved this problem

  1. Reduced version: pip install --upgrade --force-reinstall pip==9.0.3
  2. Tried to re-install package: pip install xxx --disable-pip-version-check
  3. At last, recover the latest version for pip: pip install --upgrade pip

it works for me， Thanks!

+1
anancds on Apr 8, 2019

by upgrading to newer version it should solve the problems but pip introduces problems what a irony.

+1
hemangjoshi37a on Jan 19, 2019

My reading of this is that we should be using virtual environments to protect ourselves from oddness in our environments… however, I was able to get past this for now by adding the --user flag:

pip3 install -r requirements.txt --user

https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-user

+1
pherris on Dec 6, 2018

This “not our problem, someone else fix it” is to be expected from pip. The software doesn’t even pretend to manage package conflicts. I got this message as part of an ansible deployment of containers to many systems.

Thank you for those of you who supplied workarounds. I’ve changed the ordering - “pip install docker” prior to “pip install --upgrade pip” - worked this time. Hopefully this won’t cause problems (such as data corruption) in future.

+1
hanc69 on Dec 2, 2018

@AddoSolutions the problem there is not pip, but other packages - pip simply stopped pretenses and dropped support for really bad things, and now stuff falls apart on bad actors like enterprise distros - so you have to sort out the package in terms of the enterprise distros and with your vendor

just a FYI - you will likely be told to use pip from the system (managed by the vendor) and not a random latest version

+1
RonnyPfannschmidt on Nov 22, 2018

wrt the updates of the yum package - that’s pretty much the job of the distribution - and it hasn’t been doing it, but neither have the packages themselves in some cases

when it comes to enterprise distros - things are so painfully outdated - sort it out with your vendor - open source upstreams have no power over that and should never have to bear the support burden for your choice in enterprise distros - so go to your vendor

+1
RonnyPfannschmidt on Nov 22, 2018

@pfmoore Thanks for the quick answer!

This is rather inconvenient because most of the packages come installed as dependencies and we do not manage it ourselves. Automating the removal will be interesting.

I see there is some movement to only upgrade packages, without uninstalling them first. It would be great if this would eventually happen.

We could close this issue since it is widely discussed somewhere else. Thanks!

+1
ramonamis on Apr 16, 2018
Read more comments on GitHub
← pip: Upgrading pip fails on Windows when install path is too long
pip: Using --target with --editable results in "internal" error →