setuptools: [BUG] `pip install -e` broken since setuptools==60.0.0
setuptools version
Since setuptools==60.0.0
Python version
3.7.13
OS
Ubuntu 18.04
Additional environment information
No response
Description
I have a python library in local, which I’m developing. Therefore, I’m interested in installing it with pip install -e
as I’ve always done, to develop it while it’s installed.
However, today I updated setuptools
from 46.1.3
to the newest (62.1.0
) and I realized that even if pip install -e /path/to/my/lib
reports no error in output, but I cannot import the library in python.
I’ve tracked down the version that broke this and it looks like it’s 60.0.0
.
Install setuptools==59.8.0
:
root@7a13ec49fbba:~# pip install --upgrade setuptools==59.8.0
Collecting setuptools==59.8.0
Using cached setuptools-59.8.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-59.8.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
Install library with pip install -e
:
root@7a13ec49fbba:~# pip install -e work/ner
Obtaining file:///root/work/ner
Installing collected packages: ner
Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
Try to import library in python:
root@7a13ec49fbba:~# python3
Python 3.7.13 (default, Apr 24 2022, 01:04:09)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
>>> ner.__version__
'1.0.0'
It works with setuptools==59.8.0
.
Now let’s do the same with setuptools==60.0.0
:
root@7a13ec49fbba:~# pip install --upgrade setuptools==60.0.0
Collecting setuptools==60.0.0
Downloading setuptools-60.0.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-60.0.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
Install library with pip install -e
:
root@7a13ec49fbba:~# pip install -e work/ner
Obtaining file:///root/work/ner
Installing collected packages: ner
Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
Try to import library in python:
root@7a13ec49fbba:~# python3
Python 3.7.13 (default, Apr 24 2022, 01:04:09)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ner'
Expected behavior
The installed library via pip install -e
should be able to be imported like with setuptools < 60
How to Reproduce
- Install setuptools>60 via
pip install setuptools==60.0.0
- Download a library to install it in local. e.g. via
git clone https://github.com/ivsanro1/gft-ner
- Try to install library via
pip install -e
. e.g.pip install -e ~/repos/gft-ner
- Try to import the library in python, via
python3 -c "import ner"
Output
root@7a13ec49fbba:~# pip install setuptools==60.0.0
Collecting setuptools==60.0.0
Using cached setuptools-60.0.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-60.0.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@7a13ec49fbba:~# cd repos/
root@7a13ec49fbba:~/repos# git clone https://github.com/ivsanro1/gft-ner
Cloning into 'gft-ner'...
remote: Enumerating objects: 60, done.
remote: Counting objects: 100% (60/60), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 60 (delta 21), reused 60 (delta 21), pack-reused 0
Unpacking objects: 100% (60/60), done.
root@7a13ec49fbba:~/repos# cd ..
root@7a13ec49fbba:~# pip install -e repos/gft-ner
Obtaining file:///root/repos/gft-ner
Installing collected packages: ner
Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@7a13ec49fbba:~# python3 -c "import ner"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'ner'
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 5
- Comments: 21 (12 by maintainers)
Commits related to this issue
- Workaround for setuptools bug on editable eggs See pypa/setuptools#3301 — committed to corpusops/bitwardentools by kiorky 2 years ago
- Fix an issue with ubuntu, distutils, and pip The issue appears when doing 'pip install -e <package>', the package does not show up when doing 'pip freeze'. This is caused by Ubuntu installing a patc... — committed to girder/girder by manthey a year ago
I don’t think there is much we can do from the
pypa/distutils
/pypa/setuptools
side in this issue.The Debian maintainers seem to be aware that
_distutils_system_mod.py
is required to interoperate with global installations using the latest versions ofsetuptools
(as they do implement it forPython3.10
).Meanwhile, users can do the following to workaround of this issue:
Thank you very much @ivsanro1 for re-checking that. I forgot to jump outside of the directory in the last example 😝 .
Between v60 and v61 there have been a series of changes imported from
pypa/distutils
, which might not be compatible with the patches provided by Debian/deadsnakes.I tried these steps:
Using the following commands we can see that the
ner
package is installed to/usr/lib/python3.7/site-packages/
.What seems to be happening here is the following:
dist-packages
We can see that they introducedeb_system
andunix_local
installation schemes. ~However it seems to be necessary to explicitly pass the--install-layout=deb
parameter to tap into this behaviour. This parameter is not available for editable installs and not exactly meant for usage with pip.~ The equivalentsite-packages
folder is not added tosys.path
. ~Maybe this is done on purpose to prevent users installing packages globally (which could be a way Debian maintainers can prevent users accidentally crashing the system beyond repair).~ UPDATE: The patch seems to select theunix_local
installation scheme by default, which should map to/usr/local/lib/python3.7/dist-packages
and therefore is available onsys.path
(there is no attempt to prevent users for installing packages globally).setuptools/_distutils
instead of Python’s standard library, by default. Therefore, no patch is applied to thesetuptools/_distutils
code, and, as a consequence, the packages are installed by default tosite-packages
.pypa/distutils
andpypa/setuptools
created a mechanism for Debian to customize the installation layout without the need of patches: they need to add a_distutils_system_mod.py
file to the standard library folder.