poetry-plugin-export: Dependency walk failed with 2 different version of same package

For the urllib3 workaround, you can add something like:

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.31.0"
nltk = "^3.8.1"
boto3 = "^1.28.62"
# workaround for: https://github.com/python-poetry/poetry-plugin-export/issues/183
urllib3 = ">=1.25.4,<1.27"

I just ran into this because botocore has different range requirements depending on the python version:

[[package]]
name = "botocore"
version = "1.31.62"
description = "Low-level, data-driven core of boto 3."
optional = false
python-versions = ">= 3.7"
files = [
    {file = "botocore-1.31.62-py3-none-any.whl", hash = "sha256:be792d806afc064694a2d0b9b25779f3ca0c1584b29a35ac32e67f0064ddb8b7"},
    {file = "botocore-1.31.62.tar.gz", hash = "sha256:272b78ac65256b6294cb9cdb0ac484d447ad3a85642e33cb6a3b1b8afee15a4c"},
]

[package.dependencies]
awscrt = {version = "0.16.26", optional = true, markers = "extra == \"crt\""}
jmespath = ">=0.7.1,<2.0.0"
python-dateutil = ">=2.1,<3.0.0"
urllib3 = [
    {version = ">=1.25.4,<1.27", markers = "python_version < \"3.10\""},
    {version = ">=1.25.4,<2.1", markers = "python_version >= \"3.10\""},
]

[[package]]
name = "urllib3"
version = "1.26.17"
description = "HTTP library with thread-safe connection pooling, file post, and more."
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
files = [
    {file = "urllib3-1.26.17-py2.py3-none-any.whl", hash = "sha256:94a757d178c9be92ef5539b8840d48dc9cf1b2709c9d6b588232a055c524458b"},
    {file = "urllib3-1.26.17.tar.gz", hash = "sha256:24d6a242c28d29af46c3fae832c36db3bbebcc533dd1bb549172cd739c82df21"},
]

[package.extras]
brotli = ["brotli (==1.0.9)", "brotli (>=1.0.9)", "brotlicffi (>=0.8.0)", "brotlipy (>=0.6.0)"]
secure = ["certifi", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "ipaddress", "pyOpenSSL (>=0.14)", "urllib3-secure-extra"]
socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]

[[package]]
name = "urllib3"
version = "2.0.6"
description = "HTTP library with thread-safe connection pooling, file post, and more."
optional = false
python-versions = ">=3.7"
files = [
    {file = "urllib3-2.0.6-py3-none-any.whl", hash = "sha256:7a7c7003b000adf9e7ca2a377c9688bbc54ed41b985789ed576570342a375cd2"},
    {file = "urllib3-2.0.6.tar.gz", hash = "sha256:b19e1a85d206b56d7df1d5e683df4a7725252a964e3993648dd0fb5a1c157564"},
]

[package.extras]
brotli = ["brotli (>=1.0.9)", "brotlicffi (>=0.8.0)"]
secure = ["certifi", "cryptography (>=1.9)", "idna (>=2.0.0)", "pyopenssl (>=17.1.0)", "urllib3-secure-extra"]
socks = ["pysocks (>=1.5.6,!=1.5.7,<2.0)"]
zstd = ["zstandard (>=0.18.0)"]

$ poetry export --no-interaction     --no-ansi     --only main     --format requirements.txt

The `urllib3` package has the following compatible candidates `[Package('urllib3', '1.26.17')]`;  but, the exporter dependency walker previously elected `urllib3 (2.0.6)` which is not compatible with the dependency `urllib3 (>=1.25.4,<1.27)`. Please contribute to `poetry-plugin-export` to solve this problem.
$ poetry --version
Poetry (version 1.6.1)

I am using Poetry 1.7.0 and Python 3.9 and I got this problem today with the following message:

The `urllib3` package has the following compatible candidates `[Package('urllib3', '1.26.18')]`;  
but, the exporter dependency walker previously elected `urllib3 (2.0.7)` 
which is not compatible with the dependency `urllib3 (>=1.25.4,<1.27)`. 
Please contribute to `poetry-plugin-export` to solve this problem.

This workaround works for me. Thanks @ekaj2!

For the urllib3 workaround, you can add something like:

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.31.0"
nltk = "^3.8.1"
boto3 = "^1.28.62"
# workaround for: https://github.com/python-poetry/poetry-plugin-export/issues/183
urllib3 = ">=1.25.4,<1.27"

I’m currently working around this by doing:

poetry install --only main --sync
poetry run pip freeze > requirements.txt

poetry install --sync
poetry run pip freeze > dev_requirements.txt

I’m curious to hear how poetry-plugin-export’s output would differ from this (if it were working).

So the export fails when there’s 2 different versions of urllib3, but both versions are placed into the lockfile by Poetry’s dependency resolution. Is that valid/intended? Is that a separate Poetry issue? If so, is it already being tracked on that project? It feels problematic to have 2 different versions of the same package locked and installed.

It’s intended and completely valid. The lockfile is environment-independent. This is purely an issue of poetry-plugin-export. Although two versions are locked only one is installed.

Note: As a workaround, I added an additional dependency grpcio<=1.49.1 such that 1.51.3 is not added to the .lock file. The export works fine in this case.

I’m encountering this while trying to use sentence-transformers while having to manually specify the torch packages to use in order to get the CPU version of torch.

Minimal reproduction here: https://gist.github.com/colindean/2d9e042ae92a8df86bec4e83b4ce9cc4

For the urllib3 workaround, you can add something like:

That seems to work, though the exact specification could change depending on your python version (boto says it’s fine with urllib3 2.0.x for Python 3.10 and up)

So the export fails when there’s 2 different versions of urllib3, but both versions are placed into the lockfile by Poetry’s dependency resolution. Is that valid/intended? Is that a separate Poetry issue? If so, is it already being tracked on that project? It feels problematic to have 2 different versions of the same package locked and installed.

@radoering How would you recommend we solve this within the export plugin? Should there be a flag to only make the requirements.txt for the current environment?

The workaround of forcing older version of urllib3 for all downstream libraries is not a viable long-term solution, IMHO.