pip: pip v10 breaks Debian/Ubuntu pip3 command

We solved this issue by clear hash in bash:

$ hash -d pip

Or in dash (sh):

$ hash -r pip

Also hit this issue while building docker images.

@RonnyPfannschmidt says “replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout”, which has 6 thumbs up. I find this to be an especially obtuse comment given that I instructed to do so by pip itself:

You are using pip version 8.1.1, however version 10.0.0 is available. You should consider upgrading via the ‘pip install --upgrade pip’ command.

If there is some validity to that comment, then the creators of pip should remove this message and I would encourage @RonnyPfannschmidt to raise an issue to that effect and make his case.

on extra note - replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout

Thanks, hadn’t realized that replacing the system pip was a bad idea but it makes sense. However, it would be good UX if pip would not nag about upgrading in that case. Would that be possible? I reckon a lot of people (including me) would just do whatever “the thing” asks them to do.

It works for me: curl https://bootstrap.pypa.io/get-pip.py | sudo python

First, some notes about what happened here on Debian/Ubuntu (and likely a few more Linux Distros):

• pip does not support the use of it’s internals by importing it. More on that in the documentation here.
• Debian (hence Ubuntu) does not support modifying their package manager managed files using something, that well, isn’t their package manager.

This issue is caused by, well, both being violated in a way.

• Debian uses the pip’s internal methods (which no longer work due to a reorganization of pip’s internals). Debian’s assuming here that the pip version in their repositories is the one that would be installed.
• Running pip install --upgrade pip, as root, without any other parameters modifies files that are supposed to managed by apt, which breaks the script by Debian.

Some general tips on Linux:

• It’s a good habit to use --user whenever outside a venv.

  pip install --upgrade --user pip
  

• Never run pip with sudo unless you know what you’re doing.

---

What’s the workaround?

@standag’s solution is useful when this is caused by bash’s caching of executables.

hash -r pip # or hash -d pip

If you’ve modified your OS package manager’s installation of pip (eg by using sudo pip) and python -m pip is still working, one workaround is to uninstall the pip installed version and reinstall the package manager installed version.

python -m pip uninstall pip  # this might need sudo
sudo apt install --reinstall python-pip

If you’re not on Debian/Ubuntu and pip broke for you, try running:

python -m pip install --force-reinstall pip

If the above this doesn’t resolve your problems, please file a new issue.

---

[edited by @pradyunsg: make it more relevant for linking everyone with similar issues to this comment; update the suggestion to include uninstalling/reinstalling workaround]

I’m sorry, but I’d like to point out that running python code curl’d from some website with root access is just terrifyingly insecure.

on extra note - replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout

Convince the debian/ubuntu people to not vendor and then let rot half their packages, and that’d be a valid argument.

@RonnyPfannschmidt

replacing the system pip using pip is always an act of system-vandalism where the one inflicting it is responsible for the fallout

is a comment of mental vandalism. As if the person doing the (naive) upgrade is intentionally setting out to damage their own installation… If that were the case, then pip itself shouldn’t be nagging the user to upgrade from 9.0.1 to 10.0.1 with every single pip command executed. I myself followed that recommendation and ended up in this mess.

Fortunately: sudo python -m pip install pip==9.0.1 was an easy enough remedy.

Blaming the victim is no answer, though.

How to reproduce this under a fresh Ubuntu 17.10 installation, including the evidence that the commands in comment 5221 fail to fix it, and what I proposed does fix it.

Installation of both pips (system and user), which breaks the pip command:

vfisa@vilos:~$ sudo apt install python-pip
(...)

vfisa@vilos:~$ pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

vfisa@vilos:~$ which pip
/usr/bin/pip

vfisa@vilos:~$ pip install pip --upgrade --user
Collecting pip
  Downloading https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 631kB/s 
Installing collected packages: pip
Successfully installed pip-10.0.1

vfisa@vilos:~$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

vfisa@vilos:~$ python -m pip --version
pip 10.0.1 from /home/vfisa/.local/lib/python2.7/site-packages/pip (python 2.7)

vfisa@vilos:~$ which pip
/usr/bin/pip

As you can see, the pip command points to the system pip by default, not the user installed one.

Commands from referenced comment, evidence of them not fixing the problem:

vfisa@vilos:~$ hash -r

vfisa@vilos:~$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

vfisa@vilos:~$ hash -d
hits	command
   1	/usr/bin/pip

vfisa@vilos:~$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

vfisa@vilos:~$ python -m pip install pip --force-reinstall --user
Collecting pip
  Using cached https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-10.0.1

vfisa@vilos:~$ pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

vfisa@vilos:~$ which pip
/usr/bin/pip

As you can see, as long as both pips are installed and the pip command points to the system pip (default behavior in Ubuntu), the problem will persist.

Fix option 1:

Remove system pip, keep user pip, which by default isn’t accessible via the pip command (so you need to use python -m pip).

vfisa@vilos:~$ sudo apt remove python-pip
(...)

vfisa@vilos:~$ pip
bash: /usr/bin/pip: No such file or directory

vfisa@vilos:~$ python -m pip --version
pip 10.0.1 from /home/vfisa/.local/lib/python2.7/site-packages/pip (python 2.7)

You could add ~/.local/bin to your PATH env var, to be able to use the pip command with the user pip.

Fix option 2:

Remove user pip, keep the system pip, which is older but by default has a working pip command in the path.

vfisa@vilos:~$ python -m pip uninstall pip
Uninstalling pip-10.0.1:
  Would remove:
    /home/vfisa/.local/bin/pip
    /home/vfisa/.local/bin/pip2
    /home/vfisa/.local/bin/pip2.7
    /home/vfisa/.local/lib/python2.7/site-packages/pip-10.0.1.dist-info/*
    /home/vfisa/.local/lib/python2.7/site-packages/pip/*
Proceed (y/n)? y
  Successfully uninstalled pip-10.0.1
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

vfisa@vilos:~$ pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

Always prefer “python3 -m pip” over pip3" or even better “/usr/bin/env python3 -m pip” it is safer and allow to avoid this issue with pip10

@fake-name additionally the general suggestion for usage on all distros is - use a virtualenv, don’t vandalize the system, and that works - people just font follow it and then wonder when stuff breaks and blame pip

there has been plenty of manual and automated testing with virtualenvs which works

also in a virtualenv at least there should be no debian made pip3 command - so what the heck are you talking about wrt this breaking in virtualenvs - please provide enough data to actually verify instead of whining about breakages without providing the data needed to verify them

pip is volunteer driven, not a company with dozens of employees

@pradyunsg in many cases the solutions given in that comment won’t work. Under a fresh Ubuntu 17.10, run pip install --upgrade pip: after that the pip command will be broken, and the solutions from the comment won’t fix it. And they shouldn’t!

Having a system installed pip 9 with a user installed pip 10, makes the system pip script try to import main() from the user pip 10, with an incorrect import path. Hash -r or -d won’t fix that because the pip command will still run the system pip by default. And neither will fix it upgrading user pip, as system pip will still be 9, user pip will still be 10, so the import will keep failing.

The solution for those cases must be uninstalling one of both pips.

• python -m pip uninstall pip --user, keeping the system pip, which is older

or

• sudo apt remove python-pip and keep the user installed pip, which won’t be accessible by running pip in the terminal by default. You will either need to run it with python -m pip, or add the paths to your PATH env var, etc.

All of this applies to both, pip under python 2 and 3.

Same issue on Fedora, too.

thats a debian issue

I found an unrelated cause of this pip v10 issue. When upgrading pip using a very old system pip (v1.5.6 on Debian Jessie, i.e. oldstable) for which --system is the default, I noticed that incorrect scripts were installed, e.g. /usr/local/bin/pip containing from pip import main – which I found by looking at the files. I assume this is because the older pip (or perhaps installation packages that it uses) install the .whl file incorrectly.

python -m pip install --force-reinstall pip fixed this.

Hey @rod-app!

If that were the case, then pip itself shouldn’t be nagging the user to upgrade from 9.0.1 to 10.0.1 with every single pip command executed.

We’ve noticed this and worked with OS vendors to avoid this in future versions of pip. – #5346.

@fisadev Sure, that’s needed to support user installable ‘pip install --user’ package. But I think it should be told to users "if you want to force the update to pip 10 before debian/ubuntu package is updated, you need to use pip install --user --upgrade pip and ensure $HOME/.local/bin is in your path. It’s simple to do.

I found pip3, version 9.0.1 installed in a virtualenv from a project and copied it to my /usr/bin and it works again. Here are the contents of the pip3 executable for those who would like to fix it themselves as well.

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

I’m sure all you have to do is save that to a file called pip3, make it executable by running sudo chmod +x ./pip3, run sudo apt remove python3-pip, and then copy it to the bin directory by running sudo cp ./pip3 /usr/bin. Here’s the raw file for those who just want to download and move it. pip3.zip

Perhaps there should be a “beta” channel, or some similar mechanism for people to do more testing prior to release, rather then just dumping a broken version on pypi and causing everyone’s builds to explode.

@fake-name There were two pre-releases made:

• 10.0.0b1 was released on Mar 31st
• 10.0.0b2 was released on April 2nd