setuptools: [BUG] setuptools 64.0.0 breaks editable install

setuptools version

setuptools==64.0.0

Python version

Python 3.10.6

OS

Ubuntu jammy

Additional environment information

No response

Description

After creating a simple project and installing it as editable with pip install -e, I’m unable to import modules from the package.

Expected behavior

We should be able to import a package installed as editable.

How to Reproduce

1. Create this simple project structure

pyproject.toml

[build-system]
requires = [
  "setuptools==64.0.0",
  "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.0.0"

[tool.setuptools]
packages.find.include = ["foo.*"]

setup.py

from setuptools import setup

setup()

foo/__init__.py

__version__ = "1.0.0"

2. Create and activate a venv:

python -mvenv venv
. venv/bin/activate

3. Install the project as editable:

python install -e .

Change to another directory and try to import the package:

python -c "import foo"

Output

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'foo'

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (11 by maintainers)

Commits related to this issue

Lock to setuptools 63 for building Testing if we're getting hit by https://github.com/pypa/setuptools/issues/3504? — committed to NickCrews/dedupe by NickCrews 2 years ago
Lock to setuptools 63 for building Testing if we're getting hit by https://github.com/pypa/setuptools/issues/3504? — committed to NickCrews/dedupe by NickCrews 2 years ago
ci: pin setuptools The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues/3504 for more details. — committed to P403n1x87/dd-trace-py by P403n1x87 2 years ago
ci: pin setuptools (#4085) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues/3504 for ... — committed to DataDog/dd-trace-py by P403n1x87 2 years ago
ci: pin setuptools (#4085) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues/3504 for m... — committed to DataDog/dd-trace-py by P403n1x87 2 years ago
ci: pin setuptools (#4085) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues/3504 for m... — committed to DataDog/dd-trace-py by P403n1x87 2 years ago
ci: pin setuptools (#4085) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues/3504 for m... — committed to DataDog/dd-trace-py by P403n1x87 2 years ago
ci: pin setuptools (#4085) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues/3504 for m... — committed to DataDog/dd-trace-py by P403n1x87 2 years ago
ci: pin setuptools (#4085) (#4088) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues... — committed to DataDog/dd-trace-py by mergify[bot] 2 years ago
ci: pin setuptools (#4085) (#4087) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues... — committed to DataDog/dd-trace-py by mergify[bot] 2 years ago
ci: pin setuptools (#4085) (#4086) ## Description The latest release of setuptools has broken support for editable installs, which is what riot does. See https://github.com/pypa/setuptools/issues... — committed to DataDog/dd-trace-py by mergify[bot] 2 years ago
Lock to setuptools 63 for building Testing if we're getting hit by https://github.com/pypa/setuptools/issues/3504? — committed to NickCrews/dedupe by NickCrews 2 years ago

Most upvoted comments

Ok, I think I understand what is happening based on the example provided in https://github.com/pypa/setuptools/issues/3504#issuecomment-1212585233. Thank you very much @ajasmin.

Deep explanation

The given package configuration is effectively telling setuptools to include in the wheel all the packages named "foo.*", but to not include the foo package itself.
In a regular installation setuptools will try to follow this configuration. However, (since packages are implemented in Python as normal directories) in order to create the foo.bar package, setuptools will add the foo/bar/__init__.py path to the wheel (which is a regular zip file).
When the wheel file gets unzipped, the directory foo/bar will be created, and therefore also the directory foo.
Since a regular directory is also a valid Python package (even if it does not include a __init__.py) a new foo namespace package gets created as a side-effect.
- This foo namespace package does not exactly correspond to the foo directory in the original project source tree:
  - For example, if you add Python files there (e.g. my_project/foo/other.py), they will be ignored and not added to the wheel.
  - Consider it as a brand new, unrelated, foo directory that gets created automatically to allow the foo/bar directory to exist.

In an editable installation, we don’t have to use directories, we can do the installation using custom import hooks… This way, it is possible that the the “side effect” of creating foo as a namespace package is not observed, and the import fails.

Next steps

Since the idea of an editable installation is to emulate how a regular installation works, this is very definitely a bug. I think I have found a solution and I am working on a PR. Thank you for the patience.

abravalheri on Aug 12, 2022