pipx: All virtualenvs break when Python version is upgraded

my simple and dirty workaround is force pipx to re-create its shared virtualenv by deleting it, and reinstall all packages:

rm -rf ~/.local/pipx/shared/
pipx reinstall-all

The current best way to fix a homebrew upgrade of the system python would be:

pipx reinstall-all

If you installed or reinstalled your packages after pipx version 0.15.0.0, then all options, package specs, and injected packages will be remembered and reinstall-all should recreate everything just like you installed them.

oh boy. a year after this issue is opened I see that there’s been no progress to resolve it without hacky workarounds 😬 in some respects I’m glad because what I thought to be the problem has been confirmed here, but I’m also not gonna worry too much about trying to make this work. I think I don’t use this tool enough to justify the effort to side step the problem every time I do a Python upgrade.

I’m on mac, and homebrew upgrades my python all the time. All my pipx packages are easily fixed when this happens by:

pipx reinstall-all

This workaround doesn’t seem too hacky to me.

Personally, I like symlinks and would prefer to keep them. I wonder instead of pipx could have a command to repair venvs after its underlying Python changes. Actually, I guess I’d probably push this upstream and find out what is the best recommendation for any project using venvs to do after a Python upgrade… and then automate that recommendation in pipx.

I ran into this also, and used something like this to repair broken venvs. I don’t think it’s necessarily the right or best way to handle the problem, or at the very least it would need to be fleshed out a bit (there’s no error handling or usage info, sloppily hardcoded to one python version, etc). But it’s something that works in a pinch and could be the bones of something smarter.

VENV_DIR="${1:-${HOME}/.local/pipx/venvs}"

# Delete any symlinks that point nowhere
find -L "$VENV_DIR" -type l | xargs rm

# Assume that a directory is a broken virtual environment if
# after the symlink cleanup:
#
# 1. bin/activate exists
# 2. bin/python does not exist
#
# And for those cases, create a fresh virtual environment.
for dir in $(find "$VENV_DIR" -type d -depth 1); do
  if [[ -f "$dir/bin/activate" && -d "$dir/lib/python3.7" && ! -f "$dir/bin/python" ]]; then
    echo "Refreshing Python virtual environment for $dir..."
    python3 -m venv $dir
  fi
done‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I think this is fundamentally similar to OP’s workaround, just operates one level lower (as a venv repair rather than a pipx reinstall).

Thanks, @integralist, this is a bug, but separate from the original issue here. Could you please start a new issue with the above comment?

In addition, could you show what version of pipx you are using? (i.e. the output from pipx --version) The syntax for reinstall-all you are using indicates your version of pipx may be very old. Currently to specify the python executable with reinstall-all the syntax is pipx reinstall-all --python python3.

It may be that your version of pipx is so old that you don’t have many of the updates that made reinstall-all work successfully in situations like these. Also updates to pipx have made it fail gracefully to missing python executables.

I use the following and this helps, I can gist this later into a shell script

basically

# add this bash_profile
export PATH="$PATH: $HOME/.local/bin"
export PIPX_HOME="$HOME/.local/pipx"
export PIPX_BIN_DIR="$HOME/.local/bin"

reinstall

ls ~/.local/pipx/venvs/ | xargs -L 1 pipx install 

The solution to this would be to convince the Homebrew people in some way to maintain a symlink for the Python major.minor version (/usr/local/bin/python3.7), and have that exposed as the path, because that’s the level at which things don’t break if you upgrade Python. Then pipx would use that and not have its paths disappear underneath it upon a bugfix update (3.7.5 -> 3.7.6) of Python, where the later version is meant to be drop-in compatible with the earlier one. A bunch of other Python software would want that, too.

I also hit this problem. All of the solutions listed here feel more like workarounds to me. Users shouldn’t have to think about this. Anyone understands why the symlink to python3.X is automatically getting expanded, and which bit of code is doing that?

assuming this is specific to homebrew, an alternative solution is to suggest users set export HOMEBREW_NO_INSTALL_CLEANUP=1. Starting from homebrew 2.0.0 this auto cleanup feature was added

details: https://discourse.brew.sh/t/when-did-brew-upgrade-and-reinstall-start-removing-versions/4054

FWIW, all virtualenvs that directly depend on a particular version of a homebrew python will break if that version of python is deleted so this problem isn’t specific to pipx

I’ve run into this a few times. My solution has been to manage a python-latest symlink that I point to from the pipx virtualenv. If pipx or any pipx-managed application ever gives me a stacktrace like above, I check the symlink and take an appropriate action.

One thing I’ve noticed (and even looked through the code a bit to try to resolve) is that either pipx or venv itself will dereference the path to the realpath. If I use my python-latest symlink to create a new tool using pipx, e.g. pipx install --python ~/.pyenv/versions/python-latest/bin/python pycowsay, the python symlink in the virtualenv’s bin will point to /Users/user/.pyenv/versions/3.7.3/Python.framework/Versions/3.7/bin/python3.7, assuming ~/.pyenv/versions/python-latest` points to 3.7.3.

I’ve been meaning to look into exactly how pipx handles this realpath resolution (or if venv does this itself), since it would be nice to have a more stable pipx that doesn’t break on random python upgrades. Admittedly, this is rare with pyenv (unlike homebrew now), but I’ve had it happen before.

Here is a python script I hacked together that will fix all your virtual environments after an upgrade. It seems to work for me, but mileage may vary, some stuff is hardcoded, etc. Use at your own risk:

import os
from functools import lru_cache
from textwrap import dedent


def find_most_recent_homebrew_python(python_base):

    dirs = [
        item
        for item in os.listdir(python_base)
        if os.path.isdir(os.path.join(python_base, item))
    ]
    print("dirs = {}".format(dirs))
    path_to_versions = os.path.join(
        python_base, sorted(dirs)[-1], "Frameworks", "Python.framework", "Versions"
    )
    current_version_link = os.path.join(path_to_versions, "Current")
    current_version = os.path.realpath(current_version_link)
    return os.path.join(path_to_versions, current_version)


@lru_cache(maxsize=None)
def find_location_under_dir(base, location):
    print(f"Searching for location {location} under {base}")
    for (root, dirs, files) in os.walk(base):
        if location in files:
            result = os.path.join(root, location)
            print(f"Found file {location} under {base} at {result}")
            return result
        if location in dirs:
            result = os.path.join(root, location)
            print(f"Found directory {location} under {base} at {result}")
            return result
    print(f"Unable to find location {location} under {base}")
    return False


def main():

    venv_dir = os.path.expanduser("~/.virtualenvs")
    python_base = "/usr/local/Cellar/python"

    python_dir = find_most_recent_homebrew_python(python_base)

    print("python_dir = {}".format(python_dir))

    for (root, dirs, files) in os.walk(venv_dir):
        # print("root = {}".format(root))
        # print("dirs = {}".format(dirs))
        # print("files = {}".format(files))
        abs_files = [os.path.join(root, f) for f in files]
        broken_links = [
            f for f in abs_files if os.path.islink(f) and not os.path.exists(f)
        ]
        for link in broken_links:
            resolved_link = os.path.realpath(link)
            print(f"Found broken link {link} pointing to {resolved_link}")
            base, fname = os.path.split(link)
            new_target = find_location_under_dir(python_dir, fname) 
            # If we found a new target, repoint the link
            if new_target:
                print(f"Repointing {link} to {new_target}")
                os.remove(link)
                os.symlink(new_target, link)
                continue
            # If we didn't, see if we can find the filename of the old link
            missing_base, missing_fname = os.path.split(resolved_link)
            new_target = find_location_under_dir(python_dir, missing_fname) 
            if new_target:
                print(f"Repointing {link} to {new_target}")
                os.remove(link)
                os.symlink(new_target, link)
                continue
            # Require manual intervention here
            else:
                msg = dedent(
                    """
                    Unable to fix broken link {link} pointing to {resolved_link}.
                    You may want to manually intervene here.
                    Press enter to continue:
                    """
                    )
                input(msg)


if __name__ == "__main__":
    main()

As virtualenv maintainer, I tend to disagree. I think the path forward here will be for pipx shim to automatically check if the home python changed (can just check the venvs pyenv.cfg home key), and if it did trigger a reinstall against the new Python (recreate venv + reinstall packages). @cs01?

I’ve just uninstalled the homebrew version of pipx due to this error. I don’t know if anyone else noticed the connection but by default pipx choses to create its virtualenvs using the python binary found by sys.executable of the python interpreter currently running pipx. Which combined with the default behaviour of homebrew means that this is going to break… quite often.

It actually surprised me that pipx wasnt using my default current shell’s python binary (set by pyenv not homebrew).

If this is going to work reliably with homebrew, it has to use a stable python target, like the system python, a python installed using the python.com installer, or one provided by pyenv. You cant swap virtualenv symlinks around reliably, ( it works sometimes, but don’t let that fool you 😉 ) and as @chrish42 pointed out, things like HOMEBREW_NO_INSTALL_CLEANUP=1 are just workarounds, (that in particular is a rather ugly one, that will leave junk around on users machines) so it would be good to have a proper fix.

Edit: Ive just done more digging and it looks like this came up before… https://github.com/pipxproject/pipx/issues/113 is still open and refers to https://github.com/pipxproject/pipx/issues/17 as why pipx uses sys.executable.

From the comments on those issues, it looks like this issue might require some careful choices about just which python is chosen, and perhaps more explicit warnings about how an environment has to be setup, such as ~/.local/bin coming before ~/.pyenv/shims in a users path for things to work correctly.

Looks like --upgrade might be they way to go based on this answer. Want to give it a try, @raxod502?

python3 -m venv --upgrade ~/.local/pipx/venvs/poetry