pip-tools: pip-compile fails on packages using poetry

pip-compile fails if a package uses poetry which is PEP 517 compliant (https://poetry.eustace.io/docs/pyproject/#poetry-and-pep-517).

Simply doing pip install pendulum does not require e.g. poetry to be installed. However, installing it made pip-compile work which I’m not sure we as consumers of the package in question should need to do.

Environment Versions

Alpine 3.8
Python version: 3.7.2
pip version: 19.0.1
pip-tools version: 3.3.1

Steps to replicate

# DOES NOT WORK

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip-compile --generate-hashes req.txt


# WORKS

FROM python:3.7.2-alpine3.8

RUN pip3 install --upgrade pip==19.0.1 setuptools pip-tools>==3.3.1

RUN echo "pendulum" > req.txt

RUN pip3 install poetry

RUN pip-compile --generate-hashes req.txt

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 20 (14 by maintainers)

Commits related to this issue

Update dependencies Add poetry due to issue with poetry, see https://github.com/jazzband/pip-tools/issues/721 — committed to Techtonica/techtonica.org by gsong 4 years ago
Update dependencies Add poetry due to issue with poetry, see https://github.com/jazzband/pip-tools/issues/721 — committed to Techtonica/techtonica.org by gsong 4 years ago

Most upvoted comments

I was getting the following error when running pip-compile on a project that installed a package from a submodule with a pyproject.toml file (the pyproject.toml was only being used to configure black, not as a replacement for setup.py).

pip._internal.exceptions.InstallationError: Command "/Users/clarkr/.pyenv/versions/3.6.3/bin/python3.6 /Users/clarkr/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/gc/lwr8t4zs14g6y7t401x8n315xkgl13/T/tmpr6_ewj7j" failed with error code 1

Updating requirementslib fixed the issue:

python3 -m pip install -U requirementslib

RobbieClarken on Jul 10, 2019

I am super late responding to this, but I’m not really sure what toggling this does for folks. It’s pretty desirable to turn on for installation itself, but can cause issues with resolution (though which issues specifically I couldn’t tell you at this point).

We have had a lot of problems making pip-tools play nicely with pip during the resolution of editable and local packages or remote files (which pip-tools does not resolve by default). I recently implemented a complete workaround for this in requirementslib which now fully leverages pep517 with a fallback to setuptools and is extremely effective at resolving these dependencies without issue.

In the specific case of this issue, I am fairly sure you will need build isolation turned on if you want pep517 builders to work for resolution, but that’s new territory. Like I explain above I am currently ignoring this question entirely and just invoking pep517 myself. The issues I’ve run into mainly revolve around pip’s temporary directory cleanup during the isolated build process, so by handling it myself I am able to achieve build isolation while bypassing the weird issues. Hope this helps.

techalchemy on Mar 25, 2019

So that should probably be an optional flag to let consumers choose between installing required packages themselves, or doing isolated builds (which will take forever if there are lots of packages using different build systems).

I guess so. Build isolation could be set on by default and an option to disable it:

  --no-build-isolation        Disable isolation when building a modern source

atugushev on Jan 25, 2019