setuptools: [BUG] `pip install -e` broken since setuptools==60.0.0

setuptools version

Since setuptools==60.0.0

Python version

3.7.13

OS

Ubuntu 18.04

Additional environment information

No response

Description

I have a python library in local, which I’m developing. Therefore, I’m interested in installing it with pip install -e as I’ve always done, to develop it while it’s installed.

However, today I updated setuptools from 46.1.3 to the newest (62.1.0) and I realized that even if pip install -e /path/to/my/lib reports no error in output, but I cannot import the library in python.

I’ve tracked down the version that broke this and it looks like it’s 60.0.0.

Install setuptools==59.8.0:

root@7a13ec49fbba:~# pip install --upgrade setuptools==59.8.0
Collecting setuptools==59.8.0
  Using cached setuptools-59.8.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-59.8.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Install library with pip install -e:

root@7a13ec49fbba:~# pip install -e work/ner
Obtaining file:///root/work/ner
Installing collected packages: ner
  Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Try to import library in python:

root@7a13ec49fbba:~# python3
Python 3.7.13 (default, Apr 24 2022, 01:04:09) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
>>> ner.__version__
'1.0.0'

It works with setuptools==59.8.0.

Now let’s do the same with setuptools==60.0.0:

root@7a13ec49fbba:~# pip install --upgrade setuptools==60.0.0
Collecting setuptools==60.0.0
  Downloading setuptools-60.0.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-60.0.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Install library with pip install -e:

root@7a13ec49fbba:~# pip install -e work/ner
Obtaining file:///root/work/ner
Installing collected packages: ner
  Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Try to import library in python:

root@7a13ec49fbba:~# python3
Python 3.7.13 (default, Apr 24 2022, 01:04:09) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ner'

Expected behavior

The installed library via pip install -e should be able to be imported like with setuptools < 60

How to Reproduce

  1. Install setuptools>60 via pip install setuptools==60.0.0
  2. Download a library to install it in local. e.g. via git clone https://github.com/ivsanro1/gft-ner
  3. Try to install library via pip install -e. e.g. pip install -e ~/repos/gft-ner
  4. Try to import the library in python, via python3 -c "import ner"

Output

root@7a13ec49fbba:~# pip install setuptools==60.0.0
Collecting setuptools==60.0.0
  Using cached setuptools-60.0.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-60.0.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@7a13ec49fbba:~# cd repos/
root@7a13ec49fbba:~/repos# git clone https://github.com/ivsanro1/gft-ner
Cloning into 'gft-ner'...
remote: Enumerating objects: 60, done.
remote: Counting objects: 100% (60/60), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 60 (delta 21), reused 60 (delta 21), pack-reused 0
Unpacking objects: 100% (60/60), done.
root@7a13ec49fbba:~/repos# cd ..
root@7a13ec49fbba:~# pip install -e repos/gft-ner
Obtaining file:///root/repos/gft-ner
Installing collected packages: ner
  Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@7a13ec49fbba:~# python3 -c "import ner"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'ner'

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 5
  • Comments: 21 (12 by maintainers)

Commits related to this issue

Most upvoted comments

I don’t think there is much we can do from the pypa/distutils/pypa/setuptools side in this issue.

The Debian maintainers seem to be aware that _distutils_system_mod.py is required to interoperate with global installations using the latest versions of setuptools (as they do implement it for Python3.10).

Meanwhile, users can do the following to workaround of this issue:

SETUPTOOLS_USE_DISTUTILS=stdlib python3.7 -m pip install -e .

Thank you very much @ivsanro1 for re-checking that. I forgot to jump outside of the directory in the last example 😝 .

Between v60 and v61 there have been a series of changes imported from pypa/distutils, which might not be compatible with the patches provided by Debian/deadsnakes.

I tried these steps:

> docker run --rm -it ubuntu:18.04 bash

apt-get update
apt-get install -y git wget python3.7 python3.7-dev python3.7-distutils
# I am using the official Debian distribution of python3.7
# to reduce the "surface" for the bug (minimise the moving parts).
# The deadsnakes distribution should not be very different...

wget https://bootstrap.pypa.io/get-pip.py -P /tmp
python3.7 /tmp/get-pip.py
# In Debian-based systems, there is a well-know issue with pip
# which makes old-versions of setuptools to leak into the build
# environment (see pypa/pip#6264), so let's uninstall and reinstall
# setuptools just in case...
python3.7 -m pip uninstall -y setuptools
python3.7 -m pip install -U pip setuptools

git clone https://github.com/ivsanro1/gft-ner /tmp/gft-ner
cd /tmp/gft-ner
python3.7 -m pip install -e .

Using the following commands we can see that the ner package is installed to /usr/lib/python3.7/site-packages/.

ls /usr/lib/python3.7/site-packages/
# easy-install.pth  ner.egg-link  # <-- installed in this folder

python3.7 -m site
# sys.path = [
#     '/tmp',
#     '/usr/lib/python37.zip',
#     '/usr/lib/python3.7',
#     '/usr/lib/python3.7/lib-dynload',
#     '/usr/local/lib/python3.7/dist-packages',  # <- `site-packages` is missing
#     '/usr/lib/python3/dist-packages',
# ]
# USER_BASE: '/root/.local' (doesn't exist)
# USER_SITE: '/root/.local/lib/python3.7/site-packages' (doesn't exist)
# ENABLE_USER_SITE: True

What seems to be happening here is the following:

  • Previously Debian was patching distutils to install things into dist-packages We can see that they introduce deb_system and unix_local installation schemes. ~However it seems to be necessary to explicitly pass the --install-layout=deb parameter to tap into this behaviour. This parameter is not available for editable installs and not exactly meant for usage with pip.~ The equivalent site-packages folder is not added to sys.path. ~Maybe this is done on purpose to prevent users installing packages globally (which could be a way Debian maintainers can prevent users accidentally crashing the system beyond repair).~ UPDATE: The patch seems to select the unix_local installation scheme by default, which should map to /usr/local/lib/python3.7/dist-packages and therefore is available on sys.path (there is no attempt to prevent users for installing packages globally).
  • Setuptools started to “adopt” distutils in anticipation for the deprecation announced by PEP 632. This means that distutils will be loaded from setuptools/_distutils instead of Python’s standard library, by default. Therefore, no patch is applied to the setuptools/_distutils code, and, as a consequence, the packages are installed by default to site-packages.
  • The maintainers of pypa/distutils and pypa/setuptools created a mechanism for Debian to customize the installation layout without the need of patches: they need to add a _distutils_system_mod.py file to the standard library folder.
  • This file exists in Python3.10, but not in Python3.7