pip: pip fails to remove __pycache__ folders and pyc files on uninstall, breaking future installs
Description
While I originally reported this issue on https://github.com/ansible/ansible/issues/79271 but we closed it as not being able to reproduce, I was finally able to find what can cause pip uninstall to perform an incomplete installations, one that would break future installations (at least as editable).
I observed that uninstall of ansible-core package had some leftovers and all of them were __pycache__ folders with some pyc files inside them. Sample below:
.
├── ansible
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
│ │ ├── __init__.cpython-311.pyc
│ │ ├── constants.cpython-311.pyc
│ │ ├── context.cpython-311.pyc
│ │ ├── release.cpython-310.pyc
│ │ └── release.cpython-311.pyc
│ ├── _vendor
│ │ └── __pycache__
│ │ └── __init__.cpython-311.pyc
│ ├── cli
│ │ ├── __pycache__
│ │ │ ├── __init__.cpython-311.pyc
│ │ │ ├── adhoc.cpython-311.pyc
│ │ │ ├── config.cpython-311.pyc
│ │ │ └── playbook.cpython-311.pyc
│ │ └── arguments
│ │ └── __pycache__
│ │ ├── __init__.cpython-311.pyc
│ │ └── option_helpers.cpython-311.pyc
│ ├── config
....
Expected behavior
pip uninstall should ensure that folders are empty on uninstall.
pip version
23.0.1
Python version
3.11.2
OS
MacOS Ventura
How to Reproduce
- create some
__pycache__folders and pyc files inside a package, usually it is enough to chdir to the package folder and run thru python one of the files as that should create the__pycache__folders. - uninstall this package
Please note that these files can endup being created regardless if you have something like PYTHONPYCACHEPREFIX=/Users/ssbarnea/.cache/cpython/ or PYTHONDONTWRITEBYTECODE=1 defined because some development tools might not use your user defined variables.
Basically it is practically impossible to prevent some tools/python-installations from creating these temporary files within the installed package folders.
Still, removing a package should ensure that the folders are emptied or at least provides a very visible (colored?) warning when leftovers are present, so pip user will not be surprised if his future reinstallation of the package might be broken.
Output
No response
Code of Conduct
- I agree to follow the PSF Code of Conduct.
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 2
- Comments: 18 (11 by maintainers)
That would be a breaking change, and would probably be incompatible with projects that intentionally install namespace packages. So at a minimum, there would need to be some way of handling namespace packages, and an extended transition period. And I think we should remember that this problem is actually extremely rare (we know of numba and pytest doing this, and that’s all as far as I am aware). So let’s not over-react in trying to protect against it.
And I have https://github.com/numba/numba/issues/9312 open in numba where based on @pfmoore’s comment I suggest that they update
RECORDwhen adding files. So maybe this issue can be closed given thatpip’s behavior seems to be indicative of upstream bugs?Before we close, though, one drastic idea I’ll throw out there – what if
pip uninstall my_packagerefused by default to uninstallmy_packageif it can’t remove all files inmy_package/, raising an error? This could be overridable with some sort of--files-remaining=error (default) | remove | retain, where the current behavior ofpipwould beretain, the new default would beerrorto prevent this zombie namespace package stuff, andremovewould be a new option that would essentiallyrmtree. If changing the default is too drastic, maybe adding the option but keeping the default asretainwould at least give people a way to work around this stuff. (I’d tell people topip uninstall --files-remaining=remove mnein our devdocs for example.)In any case, I opened https://github.com/pypa/packaging.python.org/pull/1423 to discuss adding a bit more info to
RECORDdocumentation about this stuff.We normally remove
.pycfiles associated with the.pyfiles listed inRECORD. Details are here: the comment at the top saysI don’t completely understand why the
pytestcase is leaving.pycfiles behind - is it adding untracked.pyfiles (and leaving those behind too?)Basically, the specification requires that the
RECORDfile contains a list of all files considered part of the package. And specifically,(Side note - this spec doesn’t take
__pycache__directories into account. Pip does, but the spec should probably cover__pycache__.)Based on that, if something adds files to an installed package, and does not update
RECORDto reflect that, then a standards-compliant uninstaller (like pip) will not remove those files, and as a consequence will not remove the containing directories (which are no longer “emptied by the uninstallation”. Yes, this will mean that the directory left behind will look like a namespace package to the Python import system. And yes, that’s far from ideal. But the resolution is either for whatever adds files to maintainRECORDcorrectly, or to use some other uninstallation method to clean up the untracked files before runningpip uninstall.All of which is a long-winded way of saying “yes, it’s appropriate to report this to any tool that adds files without tracking them in
RECORD” 🙂Re-reading the
pytestexample, I see this was mentioned:The
.pycfile in question ispytest_repeat.cpython-311-pytest-7.4.2.pycThat filename doesn’t even conform to the standard.pycnaming format in PEP 488, so I’m not at all sure what’s going on here. Even if pytest is doing something that’s technically within the rules, it’s breaking a bunch of common assumptions here…I managed to reproduce an instance of this issue reliably in a manner that I believe is relevant in practice. Turns out
pytestcreates somepycfiles for its plugins, and those aren’t cleaned up withpip uninstall. I would guess that this is a bug in eitherpytest(I’m not sure if they’re supposed to create files there) orsetuptoolsrather thanpipbut I’m honestly not sure. Please inform me if you know which project is responsible for this and I’ll report it there instead.To reproduce: install a pytest plugin, run pytest once, then uninstall the plugin again. In practice we have had issues with this specifically when moving from a normal install to an editable install, but the example reproduction doesn’t go that far. Note how after running pytest once, a pyc file with pytest in the name appears in the site-packages’ pycache dir. Uninstalling the plugin removes the other pyc files but not that one.
relevant versions:
From pip’s perspective this is simple: it only removes what it has installed, and the removal is successful as far as the existing
RECORDis concerned. Things that happen outside pip’s control aren’t pip’s problem.From a user perspective, the uninstallation is successful but the package is still importable and there is no clue as to why this may be the case. It’s hard to make the case that pip is responsible for untangling this, but at the same time pip is the last tool the user interacted with and is best positioned to do something about it (and the most likely to be blamed otherwise).
Some ideas:
dpkgdoes:Agreed, it would be nice to improve the situation here. The issue is how we distinguish between a case like this and a case where the user deliberately created a file they care about in the installation directory. We need to strike a balance between not deleting the user’s data and not surprising the user by stating we’ve uninstalled when we haven’t.
It’s important here to remember that the only information we have about what “uninstalling black” means, is what’s in the
RECORDfile. And with things like namespace packages, we can’t assume that the top-levelblackdirectory isn’t needed by another installed project (those Python 3.12.pycfiles could be owned by a project calledblack-hackand we’d have no way of knowing that).I don’t want to optimise for pathological cases, though - maybe we could add some sort of warning so that at least the user knows what’s happened.
Thanks a lot for the feedback. I’ll escalate the pytest case to their issue tracker then.