setuptools: Editable installs are broken after `__editable__` and `__path_hook__` changes

Hi @lonetwin, pkgutil.iter_modules does not cover the same spectrum as the importlib machinery: it is much more limited.

Based on this comment I would say that (so far) the stdlib developers have no intention the increase the scope or add features to pkgutil. The phrase used was: pkgutil is on its way out as soon as people have the time to deprecate it.

If you absolutely need to use pkgutil you can try to opt into a different editable installation mode (e.g. pip install -e . --config-settings editable_mode=strict) – I haven’t tested it myself, so I am not sure it will work.

This is likely because setuptools released a new mechanism for implementing editable installs, as part of their PEP600 support. The pdoc3 project is probably not yet aware of the new mechanism, and will need updating.

There isn’t really anything for pip to do here, so I’m closing this issue. If it turns out that there is a pip problem here, feel free to reopen it.

@pfmoore This is likely going to create a lot of havoc. Editable install is how people operate in Python community. And if all of a sudden, entire tooling around editable install is going to break, then likely we must be planning better around it.

My 2 cents. May be, instead of enforcing the new mechanism as default, this could have been an opt in to start with. Instead of fighting this out, we decided to simply pin setuptools "setuptools <= 62.6.0". While this has put us out of misery, we are now stuck in the past. FWIW, new mechanism has decided to return a Path which returns in __path_hook__ as part of namespace. While that file actually doesn’t exist on the disk.

Have you tried Hatchling?

@potiuk Hey! Airflow’s setup.py looks quite complex which intrigues me. Would you be interested in me opening a PR for a PoC switch to Hatchling (which will fix your issue)?

@ofek Airlfow is an ASF project and community makes decisions not me personally. If you want to make a POC and have some good arguments, you will have to start dicsussion at the airflow devlist https://airflow.apache.org/community/ . But you have to - in general - have a good reasoning and justification for such a change in build system. just fixing an error that we can fix without switching to another build mechanism is not good enough reason.

By having a look on the setup.py, it seems that airflow implements a custom develop command.

Thanks for the pointer @abravalheri . I will take a closer look and see if I can work it out. I was kinda suspecting that we could have been doing something that PEP 660 does not really like 😃. I will keep you posted.

Hi @abhinavsingh,

Could you provide a minimal reproducer explaining more in details the issues you are having? (Maybe it is worthy to split multiple issues in multiple reproducers).

I tried to investigate what you are describing with the following reproducer, but I was not able to find the problematic behaviour you are referring to… (Maybe I understood something wrong? Or maybe the reproducer is missing something? Or – best case scenario – the problem might have already gotten solved in the latest release?).

# docker run --rm -it python:3.10.6 bash

cd /tmp && rm -rf /tmp/proj_root
mkdir -p /tmp/proj_root
mkdir -p /tmp/proj_root/namespace/cli
echo "def run(): print('hello world')" > /tmp/proj_root/namespace/cli/__init__.py
cat <<EOF > /tmp/proj_root/pyproject.toml
[build-system]
requires = ["setuptools>=65.1.0"]
build-backend = "setuptools.build_meta"

[project]
name = "namespace.cli"
version = "42"

[project.scripts]
namespace-cli = "namespace.cli:run"
EOF
cd /tmp/proj_root
python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
cd /var
/tmp/proj_root/.venv/bin/python -c 'import namespace; print(f"{namespace.__path__=}")'
# ==> namespace.__path__=_NamespacePath(['/tmp/proj_root/namespace'])
/tmp/proj_root/.venv/bin/namespace-cli
# ==> hello world

As you can see above, the reproducer show that whenever possible setuptools will add the corresponding path entry to __path__. Cases when this does not occur are likely associated with configurations by which users explicitly opt out from certain packages or there is simply no directory in the file system for the package to be associated with…

There are a few other remarks I would like to make:

• In Python, path entries do not need to be existing locations in the file system, this is mentioned in Python docs:

  Path entries need not be limited to file system locations. They can refer to URLs, database queries, or any other location that can be specified as a string. … Most path entries name locations in the file system, but they need not be limited to this. … Entries in sys.path can name directories on the file system, zip files, and potentially other “locations” (see the site module) that should be searched for modules, such as URLs, or database queries.

  We can also demonstrate this with the following snippet:

  # docker run --rm -it python:3.10.6 bash
  python3.10 -c 'import sys; print(sys.path)'
  # ==> ['', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages']
  ls -la /usr/local/lib/python310.zip
  # ==> ls: cannot access '/usr/local/lib/python310.zip': No such file or directory
  

• Most static analysis tools (including the ones used in code editors) have limitations in terms of aspects of the Python language they cannot handle. Import hooks (which have been part of the Python language specification for years, and are listed in PEP 660 as a perfectly viable implementation for editable installs), are notably one of these limitations. This is reported in https://github.com/pypa/setuptools/issues/3518. Setuptools understand this limitation, and I am personally happy to contribute to an effort of improving this. However we do need someone to champion a design/architecture/standard that is well received in both packaging and static analysis community. I recommend anyone interested in pushing this forward to engage in the mailing list discussion, or on the thread on discourse.

• I would also recommend setuptools’ docs on editable installs. There might be some useful information there.