pip: ImportError in system pip wrappers after an upgrade

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don’t fully understand the complications involved, it doesn’t feel like pip should self-immolate like this.

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit 😕

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side. To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

I would add this advice of not upgrading your system pip in the documentation. I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page: https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:

python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip 

After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip The request to update pip in debian repo can be found here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

I wonder how npm and cpan solve this problem? In any case, I just uninstalled system pip3 and created my own:

#!/bin/sh

exec python3 -m pip "$@"

I’m the only user running it on this system anyway. Victory!

The Ubuntu’s python-pip-9.0.1 installed(via apt get python-pip) is conflicting with Latest pip-18.0 provided by pypa. So sudo rm -rf ~/.local/lib/python2.7/site-packages/(Beware!) removes the ubuntu’s python-pip and resolves the problem. Only one pip can exist on the system. Thanks @denisglotov

Also Upgrading pip doesn’t help as it does’nt overwrite the preexisting installed python-pip but instead create another copy of pip in different directory causing the conflict.

I am pretty new to coding, however I managed to fix it in a kind of odd way. My problem was that there was nothing in my /usr/bin/ folder regarding pip. to fix this I had my buddy upload the get-pip.py program to raspberry pi, from which I downloaded the code to my computer and ran it. It then over-rode the old pip program and installed the new one.

Here’s the code we used: working computer to pi: scp /whatever_your_directory_is/get-pip.py pi@pi's_address:/pi's_directory pi to my computer (the last period is important): scp pi@10.200.130.65:/pi's_directoryi/get-pip.py . to download pip: sudo python get-pip.py

Sorry if it’s weird, but that’s what got it working for me

… and to be 100% clear, this issue is intended as a thread to help users understand and debug the various complex interactions around user config, system, 3rd party and pip behaviour that result in this type of error. The intention is to help users be able to identify where the problem lies, precisely so that they can direct issue reports at the right people.

If as a result of the diagnosis, a user confirms that there’s a pip issue, then we’d welcome a new issue being raised, with a good description of the problem, and a means of reproducing it that we can replicate with just pip.

But I will say that my experience, and the evidence in this thread, is that very often the problem turns out not to be with pip, but with some other part of the “stack” - sorry if I come across as dismissive as a result, but please understand that I, and most of the other pip devs, really have seen these issues before. In the case of pycharm, I raised a similar issue with them about their not using supported means of installing packages, a long time (years) ago - to my knowledge it never got fixed, unfortunately. Hence my expectation in your case that it’s a pycharm problem, and the traceback supports this.

To fix the PATH run Command Prompt as administrator: pip install --upgrade --force-reinstall --ignore-installed pip that fixed the issue

(For ubuntu users) Final solution! No action required! You may want to upgrade pip using python3 -m pip install --user --upgrade pip But error occurs! This because new pip locates in $HOME/.local/bin but your $PATH doesn’t point it. But actually .profile have a script to load /.local/bin to path! So just logout and login again. Then problem fixed. By the way, it seems make sense to have pip packages in user level and load it via .profile because this also for user level.

@ErichBSchulz We’re aware of that - see #5346.

Maybe I should have more explicitly said “Only ever use your system package manager to upgrade your system pip”. The more detailed comment in that bullet point makes it clearer, but I’ve edited the post to make it explicit.

sudo apt-get --reinstall install python3-setuptools python3-wheel python3-pip sudo -H pip3 install --upgrade pip

Worked for me

@Akronix, please elaborate what you mean by

I notice that pip is now included inside python

because that has no correlation with

it can be easily run by running python3 -m pip

python3 -m pip should work if pip can be found in one of the sites (usually [~/.local]/lib/python*/site-packages and AFAIK pip is only bundled with Python if one install it from the python.org’s package. The package python3-pip from your is one of the providers, but I suspect that you had also installed pip from other sources, very likely to be PyPI. You should be able to inspect it a bit further by running python3 -m pip show pip.

The reason I make this comment is because I don’t want you and others to have the confusion that pip is somehow part of CPython standard library. I am nothing against the alias and I want to leave a side note that keeping python3-pip might be a good idea just in case one run pip uninstall pip by mistake (like I did quite a few times, I’m not the most careful person ever).

oh, it might be the case, that I installed pip from PyPA and I completely forgot about it. Thank you for correcting me @McSinyx .

Again, you seem to be putting words in my mouth or misunderstanding me intentionally.

I’m pointing out issues for others. Your saying I’m saying something I’m not. Is English your first language? I’m happy to translate or find a way to make myself clearer.

There are multiple issues that get these errors.

I’m not blaming pip devs for pycharm’s/jetbrains broken code.

I am blaming JetBrains devs for not actively working as software craftspeople and actively working to mitigate these issues by doing things like add workarounds to unblock people, or even just better error messages. But If the error path results are known, why not add some debugging and diagnostics or at least a message that links to this thread?

Also, A lot of people who get these errors never themselves touch pip, and thus logically they can’t be at fault, so the dismissive tone is at best rude, and at worst targeting the wrong people and making them a victim of anger about a problem they are also innocent victims of.

Jetbrains Resharper uses a bad way to manage a virtualenv and last I checked uses its own packaged tools to do some of that management; This is why MacOS virtual environments are not compatible with numpty and other things by default, why you have these issues, because Pycharm uses its own package of a tool it calls venv instead of the virtual environment toolset that should be standard. The user is never told this and so the user rightly assumes that pip is at fault when in reality you can get this exact same error trying to install things that are not pip, without even touching pip.

Should Jetbrains be a more responsible member of the python community? Yes, Absolutely. But the python community could also do more to keep them in check, and can do more to make the common errors cited in this thread a mitigated problem that can more easily be worked around or could include some form of official guidance, and that’s not happening either.

@duaneking It’s a dev error but not from the pip devs. If devs of software A write it in a way relies on internals of library B then devs of software A are to blame. The leading underscore _ should be sufficient indication for “use at your own risk”. So you indeed seem to be blaming wrong devs.

Please demonstrate the pip bug with a clean install of pip not involving pycharm. I’m happy to address pip bugs, but we can’t control when other tools use pip in ways that are explicitly unsupported (in this case, calling pip’s internal functions).

@RafalSkolasinski It’s far safer, because you know you’re getting the version of pip installed with the version of Python that you’re running. Using the executable wrappers involves

• Taking care of any PATH issues to ensure that the wrapper you expect is getting run.
• The possibility that a vendor-modified pip executable is being used rather than the one generated by the version of pip you want to run.

Many errors that we’ve seen boil down to “you aren’t actually running what you think you are running” - either because people are inadvertently running the wrong wrapper, or because they are running a wrapper that isn’t created (and hence isn’t supported) by pip. Using python -m pip avoids these.

Look at it this way - if python -m pip fixes the issue, then the problem isn’t with pip. So fixing the wrappers is going to need either changes to your environment (PATH, most likely) or fixes from your distro vendor. Whether you want to work on a “proper” fix, given that python -m pip is a workaround, is up to you.

Personally, I think people would find it far easier to just use python -m pip. But changing years of muscle memory isn’t easy, so I doubt that will happen quickly…

I ran into the same problem. It seems that there is a conflict between pip in system and pip in python site-packages for me in my ubuntu. After upgrading the pip as the pip command suggests, I removed the system pip using sudo apt remove python-pip And restarting the bash, the error goes away.

sudo apt-get --reinstall install python3-setuptools python3-wheel python3-pip sudo -H pip3 install --upgrade pip

Worked for me

This worked for me after multiple tries on other methods.

what do you think I should do?

@duaneking

I am blaming pip devs for not actively working as software craftspeople and actively working to mitigate these issues by doing things like add workarounds to unblock people

No. If pip is broken, it needs to be fixed. However if a user is using it wrong, there is nothing that needs to be done. You dont add workarounds to fix people using software incorrectly, you educate those users on how to properly use the software. In this case, that was attempted via an issue raised with jetbrains. The fact that the user (jetbrains) didn’t fix it is not pip’s problem.

#7499 has been done, to partially mitigate this problem for now. We will remove those additional wrappers at a later date, so please treat the warning that pip spews out about “old wrappers” as serious and solve the underlying issues (or at least help by identifying them and letting us know here).

@oteph There’s nothing to fix here. See the original issue description, specifically:

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip.

We can’t provide an “official” statement on how to upgrade scripts (system pip wrappers) that we didn’t provide and that we don’t maintain.

OK, cool. I think one of the problems here is that no-one knows what “best practice” is (and everyone’s quite reasonably focused on fixing their own problem, not working out a general solution), which is why we keep getting the same sort of questions. So having somewhere we can thrash out the “right” solution will be a good step forward.

It seems to me that the advice: Only ever use your system package manager to upgrade the system pip. isn’t really sufficient: commands like pip3 install pipenv will cause pip to be upgraded, stopping it (or rather, the ubuntu 16.04 wrapper) from working.

@pfmoore It’s from tensorflow, not PyCharm. See install.R; below are the two functions that look most relevant to me. And tensorflow shoots itself in the foot where it calls pip_install three times:

This code finds that the native pip 1.5.4 is older than 8.1, so it tries to upgrade 3 packages. First pip, which works because the pip script that tensorflow installed into my virtualenv, and which I reproduced above, is compatible with 1.5.4, and thus upgrades to pip 19.0.3.

Next (still because pip 1.5 is older than 8.1) it tries to upgrade wheel, but now the same supplied script ~/.virtualenvs/r-tensorflow/bin/pip is incompatible with the just-installed pip 19.0.3 from ~/.virtualenvs/r-tensorflow/local/lib/python2.7/site-packages/pip, which no longer provides main. So the second invocation of pip fails, and thus the whole installation of tensorflow!

  # function to call pip within virtual env
  pip_install <- function(pkgs, message) {
    cmd <- sprintf("%ssource %s && %s install --ignore-installed --upgrade %s%s",
                   ifelse(is_osx(), "", "/bin/bash -c \""),
                   shQuote(path.expand(virtualenv_bin("activate"))),
                   shQuote(path.expand(virtualenv_bin(pip_version))),
                   paste(shQuote(pkgs), collapse = " "),
                   ifelse(is_osx(), "", "\""))
    cat(message, "...\n")
    result <- system(cmd)
    if (result != 0L)
      stop("Error ", result, " occurred installing TensorFlow", call. = FALSE)
  }

  # upgrade pip and related utilities if its older than 8.1
  if (installed_pip_version() < "8.1") {
    pip_install("pip", "Upgrading pip")
    pip_install("wheel", "Upgrading wheel")
    pip_install("setuptools", "Upgrading setuptools")
  }

My workaround was to do the following in a shell:

   $ ~/.virtualenvs/r-tensorflow/bin/python -m pip install \
      --ignore-installed --upgrade tensorflow==1.12.0 h5py pyyaml requests \
      Pillow keras tensorflow-hub tensorflow-probability scipy

I fixed this by opening the files /usr/local/bin/pip and /usr/bin/pip and replacing the line from pip import main with from pip._internal import main.

@aktaseren Please use the code button.

@lemon-doge linux has it. Install homebrew for macs as an alternative.

Microsoft Windows [Version 6.1.7601] Copyright © 2009 Microsoft Corporation. All rights reserved.

C:\Users\Kiddy>pip install virtalenv WARNING: pip is being invoked by an old script wrapper. This will fail in a futu re version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the unde rlying issue. To avoid this problem you can invoke Python with ‘-m pip’ instead of running pip directly. ERROR: Exception: Traceback (most recent call last): File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\cli\base_command.py”, line 186, in _main status = self.run(options, args) File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\commands\install.py”, line 258, in run isolated_mode=options.isolated_mode, File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\commands\install.py”, line 604, in decide_user_install if site_packages_writable(root=root_path, isolated=isolated_mode): File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\commands\install.py”, line 549, in site_packages_writable test_writable_dir(d) for d in set(get_lib_location_guesses(**kwargs)) File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\commands\install.py”, line 549, in <genexpr> test_writable_dir(d) for d in set(get_lib_location_guesses(**kwargs)) File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\utils\filesystem.py”, line 140, in test_writable_dir return _test_writable_dir_win(path) File “C:\Users\Kiddy\AppData\Roaming\Python\Python37\site-packages\pip_intern al\utils\filesystem.py”, line 153, in _test_writable_dir_win fd = os.open(file, os.O_RDWR | os.O_CREAT | os.O_EXCL) PermissionError: [Errno 13] Permission denied: ‘c:\program files\python37\Lib \site-packages\accesstest_deleteme_fishfingers_custard_dhjxxo’ WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.

You should consider upgrading via the ‘c:\program files\python37\python.exe -m p ip install --upgrade pip’ command.

C:\Users\Kiddy>

I think a better solution is for pip to remain update agnostic. If distros won’t keep pip up to date with their packages then we must force a newer version upon that distro when running pip3 install -U pip. I seriously doubt overwriting the pip installed with a distro with a newer version will break any python dependencies a distro is relying on. One of the great benefits of pip (along with npm and others) is that you can update your packages/dependencies agnostic of distro package updates. So you can actually keep your software updated without reliance on distro package maintainers. Instead, you get updates directly from PyPi from the python package maintainers.

Current version of pip3 for Ubuntu 18.04 arm64 is too low(9.0.1), so I upgrade it with pip3 install -U pip and ran into this problem. If installed by get-pip.py, then other problems will be caused, which is annoying.

Just delete the old script wrapper in PATH after upgrade on my macos.

$ pip3 --version
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7)

$ which pip3
/usr/bin/pip3

$ /usr/local/bin/pip3 --version
pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7)

$ sudo mv /usr/bin/pip3 /usr/bin/old_pip3
$ pip3 --version
pip 20.0.2 from /Library/Python/3.7/site-packages/pip (python 3.7)

@millerbird0 Can you please add info why this fixes the issue? My question is --ignore-installed will ignore if a package is installed right? Then, --force-reinstall shall uninstall it first if already installed?

@ZoranPandovski it worked for me. When you run the script as administrator it will fix the PATH I updated the pip as a local user using --user install that is where problem started, so i how to reinstall it as administrator to fix the problem with PATH. --force-reinstall --ignore-installed otherwise it will give you an error when you try to install.

I had the same experience with this issue #5373. The only solution for me was uninstalling the pip.

sudo apt remove python3-pip sudo apt purge python3-pip

And follow the installation guide on this site https://pip.pypa.io/en/stable/installing/ Now pip works like a charm.

So to use the latest version of pip, we have to install a local (per-user) version of pip, and leave the system-installed pip untouched. This is clear and helpful advice. Thanks. I am very lucky I saw this page in time, before using sudo to install.

But … what a mess. On a system with N users, there could be N separately-installed versions of pip, some of them identical. Spasmodically, all N users will decide they should have the latest and greatest version of pip, and upgrade their copies asynchronously. Creating this situation was a massive blunder. I’m sure it’s extremely difficult to fix now, but it’s really important!

I ran into the same problem. It seems that there is a conflict between pip in system and pip in python site-packages for me in my ubuntu. After upgrading the pip as the pip command suggests, I removed the system pip using sudo apt remove python-pip And restarting the bash, the error goes away.

This solution worked for me. Restarting bash was what I was missing before. Messy!

pip3 install --user pip seems to break my system pip.

Isn’t it a good idea to install the latest version of pip in the user’s home directory?