poetry: Picking wrong platform installing a dependency with multiple constraints

I tried several combinations and none worked. Among the things I tried were the following things:

torch = [
    {url = "https://download.pytorch.org/whl/cu102/torch-1.6.0-cp38-cp38-win_amd64.whl", python = "~3.8", markers = "sys_platform == 'win32'" },
    {version = "1.6.0", markers = "sys_platform != 'win32'" }
]

torch = [
    {url = "https://download.pytorch.org/whl/cu102/torch-1.6.0-cp38-cp38-win_amd64.whl", python = "~3.8", platform = "win32" },
    {version = "1.6.0", platform = "linux || darwin" }
]

I tried both and similar variations of the two above with poetry 1.1.0, 1.1.1, 1.1.2, 1.1.3 and 1.1.4

In all cases, the first entry was always used no matter what platform I was on. So on Linux it downloaded the windows wheel. And On Windows, if you changed the order, it took the instructions for linux / mac os.

Could be clarified how exactly this should work and with what poetry versions in specific. So far it failed with all versions, so I’m slightly confused why this issue has been marked as resolved.

@leon19 try chaging it to the following (this is only a partial workaround - see below).

guildai = [
  {url = "https://files.pythonhosted.org/packages/10/ed/ffc1a004da1a272131e776f4b1d24d980b1dbb25b3bbaea4aff5ff7ef538/guildai-0.7.0rc7-cp37-cp37m-manylinux1_x86_64.whl", platform = "linux"},
  {url = "https://files.pythonhosted.org/packages/17/bc/2bc49e33d9343e689723c0b67aa45653d79fc4b5342c12bd7836aca58e7b/guildai-0.7.0rc7-cp37-cp37m-macosx_10_14_x86_64.whl", platform = "darwin"}
]

This will work if you are on a linux environment. I did notice that the generated lockfile will only contain the first specified url (in this case command will succeed since I was working on a linux machine and linux platform was specified first). If I were to change the order, it will fail again. However, when building the wheel generated the correct metadata.

Requires-Dist: guildai @ https://files.pythonhosted.org/packages/10/ed/ffc1a004da1a272131e776f4b1d24d980b1dbb25b3bbaea4aff5ff7ef538/guildai-0.7.0rc7-cp37-cp37m-manylinux1_x86_64.whl; sys_platform == "linux"
Requires-Dist: guildai @ https://files.pythonhosted.org/packages/17/bc/2bc49e33d9343e689723c0b67aa45653d79fc4b5342c12bd7836aca58e7b/guildai-0.7.0rc7-cp37-cp37m-macosx_10_14_x86_64.whl; sys_platform == "darwin"

From what I can tell, when markers are explicitly used the constraints get populated differently.

Issue 1: This pertains to the use of “url”. When duplicate dependencies are being merged, url and markers are ignored resulting in only the first entry to be used (in this case the linux url). Ideally, the dependency graph should be branched due to the existance of the platform marker and the url should be considered when merging duplicates.

Additionally, during investigation, I had a quick look at how this was being processed. It did raise a few questions (which might be separate issue(s) to be dealt with).

When markers are explicitly provided (eg: markers = "sys_platform == 'win32'", python = ">=3.6");

Issue 2: the python version specified is ignored if markers are explicitly specified (unsure if this is expected behaviour). https://github.com/python-poetry/poetry/blob/ae6d64ded40bea9019f7e1de229cd0b360d65e70/poetry/packages/package.py#L349-L359

Issue 3: sole use of markers causes solver to be stuck in an infinite loop. As an example, this works.

pypiwin32 = [
    { version = "220", platform = "win32", python = ">=3.6"},
    { version = "219", platform = "win32", python = "<3.6"}
]

However, this causes an infinite loop.

pypiwin32 = [
    { version = "220", markers = "sys_platform == 'win32' and python_version >= '3.6'"},
    { version = "219", markers = "sys_platform == 'win32' and python_version < '3.6'"}
]

@finswimmer would be good to get your thoughts on the above.

I have a similar problem. python-ldap package won’t build on Windows, so I have to use an unofficial wheel file. I wanted to tell Poetry that it should use the official package for Linux and the locally stored wheel for Windows:

python-ldap = [
    { markers = "sys_platform == 'linux'", version = "*" },
    { markers = "sys_platform == 'win32'", path="lib/python_ldap-3.2.0-cp36-cp36m-win_amd64.whl" }

but, judging from the poetry.lock file, it seems markers are then merged and just determine whether the library should be installed at all:

[[package]]
category = "main"
description = "Python modules for implementing LDAP clients"
marker = "sys_platform == \"linux\" or sys_platform == \"win32\""
name = "python-ldap"

I’ve also asked a question on StackOverflow.

It looks like this may have regressed in 1.1.4

Now poetry 1.6.1 I still failed to handle different platform versions of torch with poetry.

poetry has a good conception but finally I have to give up using it in platform sensetive projects.

@finswimmer I’m running poetry 1.1.6, and it appears to still have problems resolving multiple constraints dependencies.

Here’s my dependencies spec:

[tool.poetry.dependencies]
python = "^3.8"
torch = [
    { version = "^1.8.1", markers = "sys_platform == 'darwin'" },
    { url = "https://download.pytorch.org/whl/cu111/torch-1.8.1%2Bcu111-cp38-cp38-linux_x86_64.whl", markers = "sys_platform == 'linux'" },
]

Given the above, when I run poetry install on a Mac, it goes for the Linux wheel instead of the vanilla torch version:

$ poetry install
Creating virtualenv poetry-test in [...]/poetry-test/.venv
Updating dependencies
Resolving dependencies... (112.3s)

  SolverProblemError

  Because poetry-test depends on torch (1.8.1+cu111) which doesn't match any versions, version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Thoughts?

@abn or @eterna2 any new findings or workarounds for this issue?

I am encountering the same behavior using multiple constraints for a package using git and path dependencies based on platform.

The poetry export -f requirements.txt command seems to generate the requirements in the following way: given multiple constraints, poetry will simply use the first entry in the list as the sole dependency and attach any constraints/markers onto it via a union (markers separated by “or”) This means that regardless of if the given constraints should conflict (which is the intention of the different constraints), it will simply install the first listed dependency if it meets any of the given constraints.

From my understanding, this is not the expected behavior, as multiple constraints dependencies are meant to allow for defining different locations for retrieving a given dependency.

@sinoroc I ran:

poetry export --without-hashes > requirements.txt

Here’s an excerpt from my pyproject.toml:

[tool.poetry.dependencies]
python = "2.7.18 || ~3.7"
cryptography = [
    { version = "2.3.1", python = "<3" },
    { version = "~3.2", python = ">=3.7" },
]

I received a requirements list with flags like this:

cryptography==2.3.1; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" or python_version < "3"
cryptography==3.2.1; python_version >= "3.7"

When attempting to pip install this, I get:

ERROR: Double requirement given: cryptography==3.2.1 (from -r requirements.txt (line 27)) (already in cryptography==2.3.1 (from -r requirements.txt (line 26)), name='cryptography')