maturin: Importing mixed Rust/Python project fails when structured as recommended
The README of this project says the following:
Mixed rust/python projects
To create a mixed rust/python project, create a folder with your module name (i.e.
lib.namein Cargo.toml) next to your Cargo.toml and add your python sources there:my-project ├── Cargo.toml ├── my_project │ ├── __init__.py │ └── bar.py ├── Readme.md └── src └── lib.rsmaturin will add the native extension as a module in your python folder. When using develop, maturin will copy the native library and for cffi also the glue code to your python folder. You should add those files to your gitignore.
With cffi you can do
from .my_project import liband then uselib.my_native_function, with pyo3/rust-cpython you can directlyfrom .my_project import my_native_function.Example layout with pyo3 after
maturin develop:my-project ├── Cargo.toml ├── my_project │ ├── __init__.py │ ├── bar.py │ └── my_project.cpython-36m-x86_64-linux-gnu.so ├── Readme.md └── src └── lib.rs
This is the structure I’m using for python-adblock. The project is written entirely in Rust, but I’m using a mixed Rust/Python project in order to include Python type stubs. I have an __init__.py which imports the neccesary symbols from the native module.
However, some of my users have reported that the imports inside __init__.py fail as follows:
>>> import adblock
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/app/adblock/__init__.py", line 1, in <module>
from .adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'
or
>>> import adblock
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/app/adblock/__init__.py", line 1, in <module>
from adblock.adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'
See https://github.com/ArniDagur/python-adblock/issues/17. As you can see, the result is the same whether I use from .adblock import ... or from adblock.adblock import .... I have only managed to reproduce this when the package is installed system wide (pip3 install .) and the current working directory is the root of my project (git repository).
Does anyone know what is going on? Am I doing something different from what is described in the README.md, or is the advice in the README.md wrong in some way?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 28 (15 by maintainers)
Commits related to this issue
- no i686 Signed-off-by: Andy Lok <andylokandy@hotmail.com> — committed to tikv/client-py by andylokandy 3 years ago
- chore: moving python package to a subdirectory It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native... — committed to onekey-sec/unblob-native by vlaci a year ago
- chore: moving python package to a subdirectory It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native... — committed to onekey-sec/unblob-native by vlaci a year ago
- chore: moving python package to a subdirectory It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native... — committed to onekey-sec/unblob-native by vlaci a year ago
- chore: moving python package to a subdirectory It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native... — committed to onekey-sec/unblob-native by vlaci a year ago
- chore: moving python package to a subdirectory It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native... — committed to onekey-sec/unblob-native by vlaci a year ago
- chore: moving python package to a subdirectory It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native... — committed to onekey-sec/unblob-native by vlaci a year ago
- Building system of farsante - replace poetry by maturin - introduce Makefile for simplify life of devs - add a maturin action - refactor the project to the recommended structure - redefine dependenci... — committed to SemyonSinchenko/farsante by SemyonSinchenko 4 months ago
- [Feat][CI] Building system of farsante (#19) * Building system of farsante - replace poetry by maturin - introduce Makefile for simplify life of devs - add a maturin action - refactor the project to... — committed to MrPowers/farsante by SemyonSinchenko 4 months ago
@stusmall Sorry for the late reply, I think your hypothesis is correct. We’ve changed the default project template to include a
pythonfolder in #855 to avoid such issue.I’m running into this same issue, but I might have a shorter path to reproducing it. I’m including as much detail as possible because I might be doing something weird in here. I’m not an experienced python dev and I’m kinda winging it as I go.
cargo install -f --path .cd test-crates/cffi-mixedpython3 -m venv venv; source venv/bin/activatepip install toxtox --skip-missing-interpreters(I’m just using the system interpreter to simply things):For some extra version info just in case it helps for reproducing the issue:
What is interesting is that if I run
maturin developbefore runningtox --skip-missing-interpretersthen the test works.So before I go further, I want to repeat, I’m not a python dev. So this might be missing the point of the initial bug or misunderstanding how modules work. But it looks to me that when
from .cffi_mixed import ffi, libis run instead of importing from cffi_mixed the wheel it is opening the directory on disk cff_mixed. When this happens before maturin develop is run, it doesn’t have native.so in place, but it does afterwards and so the tests pass@omnivagant New patch: https://github.com/messense/maturin/commit/c299b71f6f3893aa7035b0a63978bbd074ec94cb
Please give it a try. I’ve opened https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/21694@omnivagant Can you try this patch instead? https://github.com/messense/maturin/commit/5ad4f178e5176891adedfdb7d6b7da351faaabdc
I was able to get tests passing with it on x86_64 macOS using a
alpine:edgelinux/386 docker container.I was able to reproduce with the target triple patch applied.
The problem is that maturin didn’t take
target_envinto account inTarget::get_shared_platform_tag(): https://github.com/PyO3/maturin/blob/ef37ef2a753557a17798a492d8b2f53920763fbd/src/target.rs#L171-L184I was testing on aarch64 platform, it incorrectly named the
.sofile toaarch64-linux-gnuwhich should beaarch64-linux-muslaccording toimp.get_suffixes()python3 -m sysconfigoutputYou can patch
src/target.rsto make it work: