scikit-build-core: Editable installs can set `__package__` wrong for an extension submodule of a package

Editable installs can set __package__ to the wrong value for an extension submodule within a package.

Steps to reproduce:

git clone https://github.com/godlygeek/scikit-build-core-bug-reproducer
cd scikit-build-core-bug-reproducer
python3 -m venv venv
source venv/bin/activate
pip install .
python -c "import demo"  # suceeds
pip install -e .
python -c "import demo"  # fails

The demo package’s __init__.py contains an assertion that the extension submodule has __package__ set correctly (to demo, the name of the package containing the extension module). When you do a regular pip install, it does, but when you do an editable pip install the assertion fails because __package__ has been incorrectly set to demo._demo instead of to demo.

The problem goes away if you edit the _demo_editable.py in the virtualenv and change:

        if fullname in self.known_wheel_files:

to

        if fullname in self.known_wheel_files:
            submodule_search_locations = None

though I assume that change would break other things. The problem is definitely caused by the submodule_search_locations being passed to importlib.util.spec_from_file_location though, and it seems to be triggered by the fact that the module’s path demo._demo matches the name of a folder in the repo demo/_demo. Note that this is a common naming convention for extension modules though, including in CPython itself (e.g. Modules/_decimal/_decimal.c in CPython).

This was found in a real world codebase, in https://github.com/bloomberg/memray/pull/673

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (1 by maintainers)

Commits related to this issue

Most upvoted comments

That’s what this does as well, since submodules won’t be named __init__.py.

FYI, your example run code can be written as a one-liner:

$ nox -s downstream -- https://github.com/godlygeek/scikit-build-core-bug-reproducer --editable -c "import demo"
nox > Running session downstream
nox > Creating virtual environment (uv) using python in .nox/downstream
nox > uv pip install build hatch-vcs hatchling
nox > uv pip install . --no-build-isolation
nox > git clone https://github.com/godlygeek/scikit-build-core-bug-reproducer .nox/downstream/tmp/https:__github.com_godlygeek_scikit-build-core-bug-reproducer --recurse-submodules
Cloning into '.nox/downstream/tmp/https:__github.com_godlygeek_scikit-build-core-bug-reproducer'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 10 (delta 0), reused 10 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (10/10), done.
nox > cd .nox/downstream/tmp/https:__github.com_godlygeek_scikit-build-core-bug-reproducer
nox > uv pip install -e.
nox > python -c 'import demo'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/henryschreiner/git/scikit-build/scikit-build-core/.nox/downstream/tmp/https:__github.com_godlygeek_scikit-build-core-bug-reproducer/src/demo/__init__.py", line 3, in <module>
    assert _demo.__package__ == "demo"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
nox > Command python -c 'import demo' failed with exit code 1
nox > Session downstream failed.

This also uses the current source for scikit-build-core so it’s easy to iterate on a solution. 😃

I’ll try to look into this tomorrow.