poetry: Poetry build does not install build dependencies (affects both install and build)
- I am on the latest Poetry version.
- I have searched the issues of this repo and believe that this is not a duplicate.
- If an exception occurs when executing a command, I executed it again in debug mode (
-vvv
option).
- OS version and name: OS X 10.15.5 (also tested in docker on ubuntu 16.04)
- Poetry version: 1.1.0b2
- Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/mdgoldberg/96a9955c8c823f2127dbc471a954b898
Issue
I created a repo with a minimal reproduction: https://github.com/mdgoldberg/poetry-cython-example
The problem here is that Cython isn’t installed prior to building/installing the root package, even though Cython is specified in build-system.requires
in the pyproject.toml. Here’s how I am reproducing the issues, for both poetry install
and poetry build
:
$ git clone git@github.com:mdgoldberg/poetry-cython-example.git
$ cd poetry-cython-example
$ poetry install -vvv
Creating virtualenv poetry-cython-example in /Users/mattgoldberg/poetry-cython-example/.venv
Using virtualenv: /Users/mattgoldberg/poetry-cython-example/.venv
Installing dependencies from lock file
Finding the necessary packages for the current system
Installing the current project: poetry-cython-example (0.1.0)
- Building package poetry-cython-example in editable mode
- Executing build script: build.py
Traceback (most recent call last):
File "/Users/mattgoldberg/poetry-cython-example/build.py", line 6, in <module>
from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
- Adding poetry_cython_example.pth to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages for /Users/mattgoldberg/poetry-cython-example
- Adding the poetry_cython_example-0.1.0.dist-info directory to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages
$
$ poetry build -f wheel -vvv
Building poetry-cython-example (0.1.0)
- Building wheel
Traceback (most recent call last):
File "build.py", line 6, in <module>
from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
CalledProcessError
Command '['/Users/mattgoldberg/poetry-cython-example/.venv/bin/python', 'build.py']' returned non-zero exit status 1.
at ~/.pyenv/versions/3.6.10/lib/python3.6/subprocess.py:311 in check_call
307│ if retcode:
308│ cmd = kwargs.get("args")
309│ if cmd is None:
310│ cmd = popenargs[0]
→ 311│ raise CalledProcessError(retcode, cmd)
312│ return 0
313│
314│
315│ def check_output(*popenargs, timeout=None, **kwargs):
$ # now install Cython and they work
$ pip install Cython
Collecting Cython
Using cached Cython-0.29.21-cp36-cp36m-macosx_10_9_x86_64.whl (1.9 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.21
$ poetry install -vvv
Using virtualenv: /Users/mattgoldberg/poetry-cython-example/.venv
Installing dependencies from lock file
Finding the necessary packages for the current system
Installing the current project: poetry-cython-example (0.1.0)
- Building package poetry-cython-example in editable mode
- Executing build script: build.py
- Adding poetry_cython_example.pth to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages for /Users/mattgoldberg/poetry-cython-example
- Adding the poetry_cython_example-0.1.0.dist-info directory to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages
$ poetry build -f wheel -vvv
Building poetry-cython-example (0.1.0)
- Building wheel
- Built poetry_cython_example-0.1.0-cp36-cp36m-macosx_10_15_x86_64.whl
Note that for the fix, all that matters is whether Cython is installed in the “current” Python environment. If I install Cython globally but then activate the virtual env (which doesn’t have Cython), then poetry install
and poetry build
still fail. Conversely, if I install Cython in the virtualenv and try to poetry install
or poetry build
from the directory but outside the virtualenv, it also fails.
Please, let me know if I’m misusing build.py
or any other features, I know some of them may not be stabilized. For now, I’m just installing Cython before running poetry build
or poetry install` in my Dockerfile, but it really seems like I shouldn’t need to, per PEP 518. Let me know if I’m misunderstanding, and thank you for developing Poetry!
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 26
- Comments: 15 (5 by maintainers)
Commits related to this issue
- Support build dependencies Resolves: #2789 — committed to abn/poetry by abn 4 years ago
- Support build dependencies Resolves: #2789 — committed to abn/poetry by abn 4 years ago
- Support build dependencies Resolves: #2789 — committed to abn/poetry by abn 4 years ago
- Remove dontPreferSetupPy & don't remove pyproject.toml ever It seems like this bandaid to avoid some build-system related overrides causes more problems than it fixes. Let's just deal with reality up... — committed to nix-community/poetry2nix by adisbladis 2 years ago
- Remove dontPreferSetupPy & don't remove pyproject.toml ever It seems like this bandaid to avoid some build-system related overrides causes more problems than it fixes. Let's just deal with reality up... — committed to nix-community/poetry2nix by adisbladis 2 years ago
- Remove dontPreferSetupPy & don't remove pyproject.toml ever It seems like this bandaid to avoid some build-system related overrides causes more problems than it fixes. Let's just deal with reality up... — committed to nix-community/poetry2nix by adisbladis 2 years ago
- Remove dontPreferSetupPy & don't remove pyproject.toml ever It seems like this bandaid to avoid some build-system related overrides causes more problems than it fixes. Let's just deal with reality up... — committed to nix-community/poetry2nix by adisbladis 2 years ago
I would like to add that this workaround no longer works in 1.2.0a2. Hopefully the 1.2.x series will address this issue properly!
@mdgoldberg,
Unfortunately, this does not work if you want to publish and distribute your package. A user will need to install build dependencies for your package manually. It is not a good way.
For example, I need to publish a library with C++ pybind11 based extension modules. The library is distributed in prebuilt wheels for many platforms. I can add pybind11 to dev dependencies for development and building wheels. But also I want to provide sdist with source code, so that a user can install it from the source. And I don’t want to require pybind11 dependency always. Only for installing from the sources. pybind11 is not required for prebuilt wheels. I need to build dependencies here, dev dependencies do not work in this case.
At least what’s working for me is adding cython to tool.poetry.dev-dependencies. That shouldn’t be necessary, i don’t think, but it’s kinda true at least.
The PR still isn’t in a public release right? I’m still seeing the Cython issue in the latest Poetry
1.1.12
.