scikit-build-core: Editable installs can set `__package__` wrong for an extension submodule of a package
Editable installs can set __package__ to the wrong value for an extension submodule within a package.
Steps to reproduce:
git clone https://github.com/godlygeek/scikit-build-core-bug-reproducer
cd scikit-build-core-bug-reproducer
python3 -m venv venv
source venv/bin/activate
pip install .
python -c "import demo" # suceeds
pip install -e .
python -c "import demo" # fails
The demo package’s __init__.py contains an assertion that the extension submodule has __package__ set correctly (to demo, the name of the package containing the extension module). When you do a regular pip install, it does, but when you do an editable pip install the assertion fails because __package__ has been incorrectly set to demo._demo instead of to demo.
The problem goes away if you edit the _demo_editable.py in the virtualenv and change:
if fullname in self.known_wheel_files:
to
if fullname in self.known_wheel_files:
submodule_search_locations = None
though I assume that change would break other things. The problem is definitely caused by the submodule_search_locations being passed to importlib.util.spec_from_file_location though, and it seems to be triggered by the fact that the module’s path demo._demo matches the name of a folder in the repo demo/_demo. Note that this is a common naming convention for extension modules though, including in CPython itself (e.g. Modules/_decimal/_decimal.c in CPython).
This was found in a real world codebase, in https://github.com/bloomberg/memray/pull/673
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (1 by maintainers)
That’s what this does as well, since submodules won’t be named
__init__.py.FYI, your example run code can be written as a one-liner:
This also uses the current source for scikit-build-core so it’s easy to iterate on a solution. 😃
I’ll try to look into this tomorrow.