pipx: Pipx managing itself is broken on windows

Describe the bug

When pipx manages itself (pipx is installed as a pipx app) on Windows, it is no longer able to uninstall itself (since #1168 was merged).

The following error message is shown:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\not_of_interest\.local\bin\pipx.exe\__main__.py", line 7, in <module>
  File "C:\Users\not_of_interest\.local\pipx\venvs\pipx\Lib\site-packages\pipx\main.py", line 914, in cli
  File "C:\Users\not_of_interest\.local\pipx\venvs\pipx\Lib\site-packages\pipx\main.py", line 274, in run_pipx_command
  File "C:\Users\not_of_interest\.local\pipx\venvs\pipx\Lib\site-packages\pipx\commands\reinstall.py", line 53, in reinstall
  File "C:\Users\not_of_interest\.local\pipx\venvs\pipx\Lib\site-packages\pipx\commands\uninstall.py", line 151, in uninstall
  File "C:\Users\not_of_interest\.local\pipx\venvs\pipx\Lib\site-packages\pipx\util.py", line 57, in rmdir
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 808, in rmtree
    return _rmtree_unsafe(path, onexc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 636, in _rmtree_unsafe
    onexc(os.unlink, fullname, err)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 634, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access denied: 'C:\\Users\\not_of_interest\\.local\\pipx\\venvs\\pipx\\Scripts\\python.exe'

The venv folder is not deleted, as the Scripts\python.exe is still inside. This also causes following installs to break (one has to delete the remaining folder manually).

How to reproduce

On Windows:

Install pipx in pipx
pipx uninstall pipx

Expected behavior

The locked (as in use) python.exe should be moved to the trash, as introduced in line 64-76 of https://github.com/pypa/pipx/commit/6e1907bc9e8f8d0741d825911e72c1f8551e1aca#diff-c69ab827b9b12c8c732a2cfaf1056304894e775d94224f94e56313bf1f7d299f.

Analysis

In #1168, the os.system(f'rmdir /S /Q "{str(path)}"') to delete the folder was replaced by shutil.rmtree(path). The first one printed an error to the stdout it still fell through to the code introduced by https://github.com/pypa/pipx/pull/718 linked in the “Expected behavior” section, which then moved the undeletable executable and thus resolved the issue. The latter one (now) just fails with the stated error.

If any of the maintainers give feedback, how to solve it (restore the os.system-call?), I’m happy to provide a PR (shouldn’t be too hard 😆).

Personal note

Tbh #1168 is a huge disaster imho. Both (little) changes introduced had bad impact on me. One is discribed here, for the other change see my comment https://github.com/pypa/pipx/pull/1186#pullrequestreview-1819058744. I do not want to blame anyone and hugely appreciate the time of everyone involved here, but this PR really did cost me some sanity now. And as the issue #1195 arising from it shows, I’m not the only one.

Thank you all for a great tool, which I really don’t want to miss.

About this issue

Original URL
State: closed
Created 6 months ago
Comments: 20 (20 by maintainers)

Most upvoted comments

For me the bottomline stays: a person had a issue with the Microsoft-Store Python and pipx (#1164; I can not really relate to the issue, as I use Microsoft-Store Python for some time now and never had problems with pipx).

To be clear, I still believe #1164 was quite severe and did warrant a fix. At the time I was able to reproduce it on four separate Windows machines including a fresh Windows 11 install in an empty VM. Like, literally: create a VM, install latest Windows 11 from official ISO, install latest Python from Microsoft Store, install pipx, pipx doesn’t work. To me that qualifies as “software is completely broken” in this particular software configuration that a substantial number of Windows users are likely to use.

I’m quite surprised you didn’t hit that problem if you are using MS Store Python (again, I was able to reproduce in an empty fresh install VM). I’m now wondering if this might be dependent on Windows version, or perhaps MS Store Python version. Or perhaps you’re using some kind of non-default config that affects how MS Store package path redirection works.

As I was very interested in why I never had this issue (and believe me, I’m well aware of the issues MS Store Python can cause due to these strange path redirection stuff), I at least looked into that quickly and found the reason (after close review of the details in your issue it was quite obvious):

In the change log of version 1.3.0 (https://github.com/pypa/pipx/releases/tag/1.3.0) we find the line “Move pipx paths to ensure compatibility with the platform-specific user directories”, which corresponds to https://github.com/pypa/pipx/pull/1001. The documentation change in https://github.com/pypa/pipx/pull/1001/files#diff-8b042e3f94ca5c59a7cd990b950aec0073ea84fe811e7b22be51158d7b180d56 exactly states, why I could (at first) not relate or reproduce you issue: as I use pipx since years, I obviously have my venvs in ~\.local\pipx and thus, pipx still uses this folder. A new installation (or if I rename my ~\.local\pipx folder) uses some location in ~\AppData\Local, which is prone to path redirection, and I can easily reproduce #1164 then.

Bottomline: #1164 is caused by #1001 (and of cause MS Store Python path redirection of AppData\Local), but the changes of #1001 were not “active” in my setup (due to the fallback to the old folder, if it exists).

guahki on Jan 28, 2024

Just for documentation:

With this new angle, I would like the core devs to rethink https://github.com/pypa/pipx/pull/1001 (at least on windows). As this discussion might further leave the original issue reported here (while still kind of related), I’m happy to open this a new issue, if you want me to.

@Gitznik moved the discussion to https://github.com/pypa/pipx/discussions/1247, thanks!

The original issue is solved by #1231, which is under review right now.

guahki on Feb 10, 2024

setup VM and install MS Store Python (I use 3.12, but version should be not really important)

create a venv and activate it

install pipx inside venv

pipx install pipx (with pipx inside venv)

deactivate venv (important, to use the pipx installed in step 4, not the one installed in step 3) (irl I would delete the venv after “bootstrapping”, but for testing it is very convenient to be able to activate and deactivate it)

pipx uninstall pipx should cause the error, as it uses the “correct” pipx (installed by pipx) to cause the error

That did it, thanks 👍

Gitznik on Feb 4, 2024

Hi @Gitznik,

I will try to give exact details on a “minimal working example” asap, but from first reading the question is: which pipx is used in step 6. You have pipx installed via pip and pipx installed in pipx, and the question is which is first in the PATH. For the issue to occur, you have to make sure to delete pipx “with itself”, i.e. with the pipx.exe installed through step 4 (not step 3). Either by executing the pipx.exe by full path or by uninstalling the pipx installed by pip in step 3 (pip uninstall pipx).

PS: without testing and just from thinking, my workflow would be as follows:

setup VM and install MS Store Python (I use 3.12, but version should be not really important)
create a venv and activate it
install pipx inside venv
pipx install pipx (with pipx inside venv)
deactivate venv (important, to use the pipx installed in step 4, not the one installed in step 3) (irl I would delete the venv after “bootstrapping”, but for testing it is very convenient to be able to activate and deactivate it)
pipx uninstall pipx should cause the error, as it uses the “correct” pipx (installed by pipx) to cause the error

guahki on Feb 4, 2024

Yes, I think we should just make #1168 work. With this additional understanding we can easily both stand by our core statements:

#1164 was “quite severe and did warrant a fix” and the state before “qualifies as “software is completely broken”” (to cite your words).

Nevertheless, #1168 as fix for this issue still “broke the --python flag on Linux (https://github.com/pypa/pipx/issues/1195), on Windows (https://github.com/pypa/pipx/issues/1205) and (just as a sideeffect) the self-managing capability of pipx on Windows” (to cite myself), and thus maybe was not a really good fix.

So, as discussed: I will also do some more investigation, but at first glance the idea presented by @Gitznik in https://github.com/pypa/pipx/pull/1232#pullrequestreview-1847524295 looks promising to me and might be the best way forward.

guahki on Jan 28, 2024

@dechamps Even without the time to review anything in detail right now, I want to immediately make sure to reiterate: I do not want to blame you for anything! Of course, your changes broke some use-cases for me and the debugging did cost (and costs) me some time. But

As you (and @Gitznik, thanks for that!) say: you are not making decisions alone, the whole system (contributor, reviewer, unit-test, …) did not work to it’s best here
my use-cases are partially edge-case (especially the use-case discussed in this issue)
It’s the risk and beauty of OSS: the risk is, somebody might break it. The beauty is, we can join and together come to a solution which hopefully is better than anything before. I’m very thankful for your humble and concerned reply and hope very much, you did not take anything I said personal (especially not the passage you quoted).

So, overall, let’s just work on finding the best solution.

All of that said, I will review all of the comments from different sides asap, but it might take some time, as real life (unfortunately) exists.

guahki on Jan 28, 2024