poetry: Can not install any poetry based project with pip from source distribution

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
OS version and name: NixOS 18.09
Poetry version: –
Link of a Gist with the contents of your pyproject.toml file: –

Issue

Hi,

this is not really a bug about how poetry works, but how poetry interacts with other python packaging ecosystem.

While working on a project of mine which uses pip inside (for now) I noticed that we can not install a package called backoff which in its recent version switched to poetry.

To reproduce the error you simply need to invoke the following command:

$ pip install backoff --no-binary :all:

Above command tries to install packages from source distribution and not from wheel (which is a requirement of my project).

Digging deeper I noticed that problem is that poetry depends on tomlkit which depends on poetry as a build tool.

I would bet that this is a bug of pip and looking at their issue tracker I found the issue being reported there: https://github.com/pypa/pip/issues/5739. If you look closed you will see that also flit was affected but they solved it by moving their dependencies around a bit.

I’m not sure what a possible solution might be but maybe vendoring of the dependencies might be a solution (eg. like pip does it). Or maybe there is a better way…

This kinda puts poetry (and actually all projects using it) as a blocker for me and prevents me from investigating further into poetry, which I really hope becomes more used in python ecosystem. This is also a reason why I’m posting this issue here, to try to bring attention to this kinds of bugs and to make poetry adoption easier.

In any case, thank you for you work.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 8
Comments: 24 (5 by maintainers)

Most upvoted comments

The upcoming release should provide a solution for this. With the next release poetry-core will become available. Applying the following patch to backoff should enable this to work with newer pip versions. There is a bug in pip==19.1.1 that does not setup the PEP517 environment correctly, hence failing --no-binary :all: builds. The following fix was verified to work with pip==20.0.2

diff --git a/pyproject.toml b/pyproject.toml
index 3469bf2..a42652d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,5 +38,5 @@ pytest-cov = "^2.6"
 pytest-asyncio = {version = "^0.10.0",python = "^3.5"}
 
 [build-system]
-requires = ["poetry>=0.12"]
-build-backend = "poetry.masonry.api"
+requires = ["poetry-core>=1.0.0a5"]
+build-backend = "poetry.core.masonry.api"

Note that this uses an alpha release of poetry-core.

abn on Apr 18, 2020

It seems like, in lieu of a fix from pip, poetry needs to do something to solve this issue. The only real solution to me seems like vendoring any of poetry’s dependencies which, in turn, use poetry for their builds. This likely also applies to any transitive dependencies of poetry too, which makes the problem not only more complicated but prone to changing as more packages switch to poetry.

I have found a workaround which works for my use-case, at least for now. I’m building wheels for everything in the poetry.lock during a build. If I pass each of the packages to --no-binary except for tomlkit, poetry, and pastel I still get builds from source for everything else. My project doesn’t depend on those packages, they’re only needed for poetry during the build. I imagine this might still end up causing a problem for any build which wants or needs to use --no-binary :all: and which also depends on any of these 3 packages.

Here’s an example of what the (insanely long) command looks like in my case (built by a script):

poetry run pip wheel -w wheels --no-cache-dir --isolated --no-binary aniso8601 --no-binary argh --no-binary attrs --no-binary boto3 --no-binary botocore --no-binary CacheControl --no-binary cached-property --no-binary cachy --no-binary certifi --no-binary chardet --no-binary cleo --no-binary Click --no-binary colorama --no-binary coverage --no-binary dnspython --no-binary docopt --no-binary docutils --no-binary ec2-metadata --no-binary eventlet --no-binary fasteners --no-binary filelock --no-binary Flask --no-binary Flask-RESTful --no-binary greenlet --no-binary html5lib --no-binary ianitor --no-binary idna --no-binary importlib-metadata --no-binary itsdangerous --no-binary Jinja2 --no-binary jmespath --no-binary jsonschema --no-binary lockfile --no-binary magicreq --no-binary MarkupSafe --no-binary monotonic --no-binary more-itertools --no-binary msgpack --no-binary nose --no-binary nosecount --no-binary packaging --no-binary pathtools --no-binary pex --no-binary pkginfo --no-binary pluggy --no-binary psutil --no-binary py --no-binary pycryptodome --no-binary pylev --no-binary pyparsing --no-binary pyrsistent --no-binary python-consul --no-binary python-daemon --no-binary python-dateutil --no-binary pytz --no-binary PyYAML --no-binary requests --no-binary requests-toolbelt --no-binary reversefold.util --no-binary s3transfer --no-binary setproctitle --no-binary shellingham --no-binary six --no-binary tabulate --no-binary tenacity --no-binary toml --no-binary tox --no-binary urllib3 --no-binary virtualenv --no-binary watchdog --no-binary webencodings --no-binary Werkzeug --no-binary zipp aniso8601==8.0.0 argh==0.26.2 attrs==19.2.0 boto3==1.9.240 botocore==1.12.240 CacheControl==0.12.5 cached-property==1.5.1 cachy==0.2.0 certifi==2019.9.11 chardet==3.0.4 cleo==0.6.8 Click==7.0 colorama==0.4.1 coverage==4.5.4 dnspython==1.16.0 docopt==0.6.2 docutils==0.15.2 ec2-metadata==1.8.0 eventlet==0.25.1 fasteners==0.15 filelock==3.0.12 Flask==1.1.1 Flask-RESTful==0.3.7 greenlet==0.4.15 html5lib==1.0.1 ianitor==0.1.0 idna==2.8 importlib-metadata==0.23 itsdangerous==1.1.0 Jinja2==2.10.1 jmespath==0.9.4 jsonschema==3.0.2 lockfile==0.12.2 magicreq==0.6.0 MarkupSafe==1.1.1 monotonic==1.5 more-itertools==7.2.0 msgpack==0.6.2 nose==1.3.7 nosecount==3.7.2.59 packaging==19.2 pastel==0.1.1 pathtools==0.1.2 pex==1.6.11 pkginfo==1.5.0.1 pluggy==0.13.0 poetry==0.12.17 psutil==5.6.3 py==1.8.0 pycryptodome==3.9.0 pylev==1.3.0 pyparsing==2.4.2 pyrsistent==0.14.11 python-consul==0.7.2 python-daemon==2.1.1 python-dateutil==2.8.0 pytz==2019.2 PyYAML==5.1.2 requests==2.22.0 requests-toolbelt==0.8.0 reversefold.util==3.0.1 s3transfer==0.2.1 setproctitle==1.1.10 shellingham==1.3.1 six==1.12.0 tabulate==0.8.5 tenacity==5.1.1 toml==0.10.0 tomlkit==0.5.5 tox==3.14.0 urllib3==1.25.6 virtualenv==16.7.5 watchdog==0.9.0 webencodings==0.5.1 Werkzeug==0.16.0 zipp==0.6.0

reversefold on Oct 2, 2019

@cs01 i think that issue you reference to is not really related to this one, except that both show that there are issues with pip and poetry.

The problem is that there is a circular dependency in poetry dependencies (poetry <-> tomlkit) which is usually not a problem since pip (or any other python package management tool) can handle this.

This becomes a problem when you are required to use source distributions (no wheels for me aka --no-binary :all: flag) and the fact that pip needs to install a package to inspect its dependencies, that means that it needs to first install build dependencies first.

So this is what happens when you run pip install backoff --no-binary :all::

before installing backoff we need to install build dependencies
poetry is a build dependency of backoff so we install it and we now see a list of poetry dependencies
tomlkit is one of the poetry dependencies, but before installing it (to retrieve its dependency list) we need to install its build dependencies
tomlkit’s build dependency is poetry and here the the “circle” gets created.

garbas on Jan 2, 2019

I have also been commenting on the pip issue https://github.com/pypa/pip/issues/5739 and thinking more on this and it seems to me like a mis-feature of pip causing this problem (although I could be convinced otherwise if there is documentation saying otherwise) that is requiring source-based installs of build-time dependencies, especially when those dependencies are already installed and available.

In the case of the command I gave above I’m not only running the pip command via poetry, in a poetry created virtualenv, but poetry is itself already installed in that virtualenv, so poetry should be already-available to pip when it comes to building packages that require poetry at build-time.

reversefold on Oct 2, 2019

The same happens when trying to build pytzdata from source. So it is not only connected to backoff.

Dockerfile:

FROM python:3.7.2-alpine

RUN pip3 install -U pip

RUN pip3 install --no-binary :all: pytzdata

Output:

#6 [3/3] RUN pip3 install --no-binary :all: pytzdata
#6       digest: sha256:a6ecf2defd61faa5cd6faf6702173cf6fb4d526b5c746a046b834ad256b416e8
#6         name: "[3/3] RUN pip3 install --no-binary :all: pytzdata"
#6      started: 2019-01-25 17:46:47.670230476 +0000 UTC
#6 1.094 Collecting pytzdata
#6 1.803   Downloading https://files.pythonhosted.org/packages/d9/27/04b261a2d24a2658159e502fb01e9579f7a42d7a1c51ca5023190dff126b/pytzdata-2018.7.tar.gz (278kB)
#6 2.251   Installing build dependencies: started
#6 33.26   Installing build dependencies: finished with status 'error'
#6 33.26   Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5db63xt1/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
#6 33.26   Collecting poetry>=0.12
#6 33.26     Downloading https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz (150kB)
#6 33.26     Installing build dependencies: started
#6 33.26     Installing build dependencies: finished with status 'done'
#6 33.26     Getting requirements to build wheel: started
#6 33.26     Getting requirements to build wheel: finished with status 'done'
#6 33.26       Preparing wheel metadata: started
#6 33.26       Preparing wheel metadata: finished with status 'done'
#6 33.26   Collecting requests<3.0,>=2.18 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz (111kB)
#6 33.26   Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/2d/c3/0a1154fe821a62b4bd3d58b5863ff5e2314181c337c9ab8202f93b3be405/jsonschema-3.0.0b1.tar.gz (119kB)
#6 33.26   Collecting pyrsistent<0.15.0,>=0.14.2 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/45/5a/a1a446eba4b5bf9f823fd863605df24327e49241d6b5c43d82b429228caa/pyrsistent-0.14.9.tar.gz (104kB)
#6 33.26   Collecting requests-toolbelt<0.9.0,>=0.8.0 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/86/f9/e80fa23edca6c554f1994040064760c12b51daff54b55f9e379e899cd3d4/requests-toolbelt-0.8.0.tar.gz (196kB)
#6 33.26   Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
#6 33.26     Downloading https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
#6 33.26     Installing build dependencies: started
#6 33.26     Installing build dependencies: finished with status 'error'
#6 33.26     Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0h7u3yj5/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3:
#6 33.26     Collecting poetry>=0.12a3
#6 33.26       Downloading https://files.pythonhosted.org/packages/93/8a/270aa699a07405df9b260f9192eec043cf486b4b49fafd7ebcee09cb2cc5/poetry-1.0.0a2.tar.gz (154kB)
#6 33.26       Installing build dependencies: started
#6 33.26       Installing build dependencies: finished with status 'done'
#6 33.26       Getting requirements to build wheel: started
#6 33.26       Getting requirements to build wheel: finished with status 'done'
#6 33.26         Preparing wheel metadata: started
#6 33.26         Preparing wheel metadata: finished with status 'done'
#6 33.26     Collecting cleo<0.8.0,>=0.7.2 (from poetry>=0.12a3)
#6 33.26       Downloading https://files.pythonhosted.org/packages/dc/53/0782ca2ad1c38114e54b4ef9894ed3055ece8c886091606a75d43b161fb0/cleo-0.7.2.tar.gz
#6 33.26       Installing build dependencies: started
#6 33.26       Installing build dependencies: finished with status 'error'
#6 33.26       Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c_a34ziu/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
#6 33.26       Collecting poetry>=0.12
#6 33.26         Using cached https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz
#6 33.26         Installing build dependencies: started
#6 33.26         Installing build dependencies: finished with status 'done'
#6 33.26         Getting requirements to build wheel: started
#6 33.26         Getting requirements to build wheel: finished with status 'done'
#6 33.26           Preparing wheel metadata: started
#6 33.26           Preparing wheel metadata: finished with status 'done'
#6 33.26       Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/cb/9f/8dd57785485b2a2746edbe598a31b0852740a1ff91a2739c104b628c8520/cleo-0.6.8.tar.gz (60kB)
#6 33.26         Installing build dependencies: started
#6 33.26         Installing build dependencies: finished with status 'done'
#6 33.26         Getting requirements to build wheel: started
#6 33.26         Getting requirements to build wheel: finished with status 'done'
#6 33.26           Preparing wheel metadata: started
#6 33.26           Preparing wheel metadata: finished with status 'done'
#6 33.26       Collecting pyparsing<3.0,>=2.2 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/b9/b8/6b32b3e84014148dcd60dd05795e35c2e7f4b72f918616c61fdce83d27fc/pyparsing-2.3.1.tar.gz (596kB)
#6 33.26       Collecting shellingham<2.0,>=1.1 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/04/47/c3f0b5fce44827d011f7f4b8831dd20984a228d7a7f46d4d45f1aaa6fd90/shellingham-1.2.8.tar.gz
#6 33.26       Collecting cachy<0.3,>=0.2 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/68/ef/eabaf6c36bb87e4c1a74790b19e44064df25e09a55f191eb71ccd2b512ad/cachy-0.2.0.tar.gz
#6 33.26         Installing build dependencies: started
#6 33.26         Installing build dependencies: finished with status 'done'
#6 33.26         Getting requirements to build wheel: started
#6 33.26         Getting requirements to build wheel: finished with status 'done'
#6 33.26           Preparing wheel metadata: started
#6 33.26           Preparing wheel metadata: finished with status 'done'
#6 33.26       Collecting html5lib<2.0,>=1.0 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/85/3e/cf449cf1b5004e87510b9368e7a5f1acd8831c2d6691edd3c62a0823f98f/html5lib-1.0.1.tar.gz (252kB)
#6 33.26       Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
#6 33.26         Using cached https://files.pythonhosted.org/packages/2d/c3/0a1154fe821a62b4bd3d58b5863ff5e2314181c337c9ab8202f93b3be405/jsonschema-3.0.0b1.tar.gz
#6 33.26       Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12)
#6 33.26         Downloading https://files.pythonhosted.org/packages/6c/04/fd6683d24581894be8b25bc8c68ac7a0a73bf0c4d74b888ac5fe9a28e77f/pkginfo-1.5.0.1.tar.gz
#6 33.26       Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
#6 33.26         Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
#6 33.26       Exception:
#6 33.26       Traceback (most recent call last):
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
#6 33.26           status = self.run(options, args)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 315, in run
#6 33.26           resolver.resolve(requirement_set)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
#6 33.26           self._resolve_one(requirement_set, req)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
#6 33.26           abstract_dist = self._get_abstract_dist_for(req_to_install)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
#6 33.26           self.require_hashes
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 348, in prepare_linked_requirement
#6 33.26           with self.req_tracker.track(req):
#6 33.26         File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
#6 33.26           return next(self.gen)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 86, in track
#6 33.26           self.add(req)
#6 33.26         File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 55, in add
#6 33.26           % (link, fp.read()))
#6 33.26       LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit<0.6.0,>=0.5.1 from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from poetry>=0.12)
#6 33.26   
#6 33.26       ----------------------------------------
#6 33.26     Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-c_a34ziu/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 2 in None
#6 33.26   
#6 33.26     ----------------------------------------
#6 33.26   Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-0h7u3yj5/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3" failed with error code 1 in None
#6 33.26   
#6 33.26   ----------------------------------------
#6 33.28 Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5db63xt1/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in None
#6    completed: 2019-01-25 17:47:21.288965909 +0000 UTC
#6     duration: 33.618735433s
#6        error: "executor failed running [/bin/sh -c pip3 install --no-binary :all: pytzdata]: exit code: 1"

eliasbrange on Jan 25, 2019

I am the maintainer of backoff and I am trying figure out if I need to do a workaround on my side or if this is something that I can expect to be fixed upstream.

I believe the issue is that there is a circular dependency: poetry and tomlkit depend on each other at build-time. It is not clear to me if this is a pip bug, or if circular dependencies just aren’t supported when building from source.

I think @garbas is correct that vendoring (probably tomlkit in poetry?) would work around the problem. If there is a possible fix on the pip side, it sounds like it hasn’t happened yet.

Any thoughts on how to address this or if I should be looking for a workaround downstream?

bgreen-litl on Dec 30, 2018