poetry: Dataclasses solver failure with conditional python marker

As the original submitter, I can still reproduce this on the latest Poetry (1.0.9), when running poetry add <anypackage> within a Python 3.8 venv.

For more context, I’m building an application that runs on any Python from >= 3.6.1, and < 4.0.0 that requires the dataclasses library. As dataclasses is only required when developing using / when a user is running on Python >= 3.6.1 and < 3.7.0, I’ve set a conditional marker - however I develop myself within a Python 3.8 venv. I don’t believe that this is a user error, as I imagine Poetry supports such a configuration (Poetry’s own pyproject.toml has a similar dataclasses constraint) so I’m not sure what’s not working here.

The only way I can get it to work right now is by commenting the dataclasses package in my pyproject.toml, adding the additional package I want via poetry add, and then uncommenting dataclasses again.

My pyproject.toml is,

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = {version = "0.7", python = "~3.6.1"}

The output from running poetry add <anypackage> -vvv is

   1: fact: dataclasses (0.7) requires Python >=3.6, <3.7
   1: conflict: dataclasses (0.7) requires Python >=3.6, <3.7
   1: ! dataclasses (0.7) is satisfied by dataclasses (0.7)
   1: ! which is caused by "datapane depends on dataclasses (0.7)"
   1: ! thus: version solving failed
   1: Version solving took 0.420 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
The current project's Python requirement (^3.6.1) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because datapane depends on dataclasses (0.7) which requires Python >=3.6, <3.7, version solving failed.

Having a lot of trouble with this. If a dependency itself depends on a restricted version of dataclasses you also get this error, even if you don’t depend on dataclasses yourself.

Similar issue:

[SolverProblemError]
Because {some-lib} ({some-ver}) depends on dataclasses (^0.6) which doesn't match any versions, {some-lib} is forbidden.

where {some-lib} uses a pyproject.toml with

[tool.poetry.dependencies]
python = ">=3.6.1, <3.9.0"  # see tox below too
dataclasses = {version = "^0.6", python = "^3.6.1, <3.7"}

So, what’s up with dataclasses (^0.6) which doesn't match any versions?

Also, there are some PyPi packages that have started to declare minimal python versions of 3.6.1 and this seems to force the consumer to also use that minimal version, but then another issue arises like:

[SolverProblemError]
The current project's Python requirement (>=3.6.1, <3.9.0) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

An example of a 3rd-party lib causing conflict with the minimal python version when the project is trying to use python = ">=3.6, <3.9.0", which should be OK with dataclasses, is the following.

[tool.poetry.dependencies]
python = ">=3.6, <3.9.0"  # see tox below too
dataclasses = {version = "^0.6", python = "^3.6.1, <3.7"}

[tool.poetry.dev-dependencies]
pre-commit = "^2.4"

But then:

[SolverProblemError]
The current project's Python requirement (>=3.6, <3.9.0) is not compatible with some of the required packages Python requirement:
  - pre-commit requires Python >=3.6.1
  - pre-commit requires Python >=3.6.1
  - pre-commit requires Python >=3.6.1
  - pre-commit requires Python >=3.6.1

This is impossible. My {some-lib} is allowed to be used in python >= 3.6, < 4.0 and I expect to be able to use dataclasses anytime that python is a 3.6.x version (and since pre-commit is only a dev-dep, it will never be installed when {some-lib} is installed, so dataclasses trumps pre-commit, and if the actual python is 3.6.0 I don’t care if pre-commit cannot be installed), so the following solver error is some kind of a bug:

[SolverProblemError]
The current project's Python requirement (>=3.6, <3.9.0) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

So long as the dataclasses is restricted to a subset of the project python versions (i.e. the project python versions subsume all the versions allowed by dataclasses), it should pass OK. Consider even the extreme case where all of the dataclasses versions are entirely outside the range of versions allowed by the project (i.e. no version intersection), it could still allow the inclusion of the dataclasses dependency and issue a warning that it is never going to be installed anywhere that the project is allowed. Whenever there is any version intersection between the project python versions and the dependency python versions, that means the dependency could be installed and should be allowed to pass without warning or error.

Somehow, poetry is using too-tight coupling of project python versions with dependency versions. Something about the sets of allowed dependency intersections needs to be relaxed for the package-spec. The actual package installation rules are another matter entirely, because the range of options are restricted by the actual python version involved at the time of installation.

In the case of dataclasses, there is an implicit, not explicit, understanding that the package is not required for any python >= 3.7.0, so when the project declares that it can be used in a range of pythons > 3.6 but it only needs dataclasses for 3.6, it’s like an implicit declaration that dataclasses is a “required option” only for py 3.6. Somehow the version-specs and the “optional” or “extra” specs are not fully capturing this implicit intention. i.e.

dataclasses = {version = "*", python = "~3.6.0", optional = true}

It might be cool if we could declare optional = true, required = "~3.6.0" or something to suggest that this package is normally optional but when the installation is on py3.6, then it is required.

IMO, maybe python should be fully backporting dataclasses into all current py3.6 releases, rather than providing this backport as a package. Give py3.6 this little extra battery that should be included.

In case it might help, python, dataclasses and pre-commit seem to play OK together using:

[tool.poetry.dependencies]
python = "^3.6"
dataclasses = {version = "*", python = "~3.6.0"}

[tool.poetry.dev-dependencies]
pre-commit = "*"

This combination of specs was arrived at by a combination of poetry commands and manual edits. When dataclasses version is specified using ^0.6, it seems to install only 0.6 and not allow an update to 0.7. (For consumers of this lib with these specs, there might be a problem with the * version for dataclasses, TBD.)

Now without the additional package, just drop the patch version for dataclases.

[tool.poetry]
name = "test_poetry"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = { version="^0.7", python="^3.6, <3.7" }

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Reproducible in 1.0.10.

$ poetry@1.0.10 add --dry-run arrow
Using version ^0.16.0 for arrow

Updating dependencies
Resolving dependencies... (0.0s)

[SolverProblemError]
The current project's Python requirement (^3.6.1) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because dataclasses (0.7) requires Python >=3.6, <3.7
 and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
So, because poetry-demo depends on dataclasses (^0.7), version solving failed.

This should be fixed in 1.1.0b2.

$ poetry@1.1.0b2 add --dry-run arrow
Using version ^0.16.0 for arrow

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 0 updates, 0 removals, 5 skipped

  • Removing dataclasses (0.7): Skipped for the following reason: Not currently installed
  • Installing six (1.15.0): Skipped for the following reason: Already installed
  • Installing python-dateutil (2.8.1): Skipped for the following reason: Already installed
  • Installing arrow (0.16.0): Skipped for the following reason: Already installed
  • Installing dacite (1.5.1): Skipped for the following reason: Already installed

Same problem as @mands

• Define a project that needs python >=3.6 / 3.6.1 / ^3.6 / ^3.6.1
• poetry add dataclasses --python ">=3.6, <3.7"
• installs fine
• then add any other package poetry add --dev black
• fails with

[SolverProblemError]
The current project's Python requirement (^3.6) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because dataclasses (0.7) requires Python >=3.6, <3.7
 and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
So, because betterproto depends on dataclasses (^0.7), version solving failed.

I can confirm the workaround above works. (Remove dataclasses, add new package, re-add dataclasses)

@xylix see #2950

This happens when adding black ^20.8b1 to dev requirements.

Unfortunately, I have to install black separately because of this issue 😦

Poetry could print a hint or offer to fix the project requirements with one of the available options that would allow to proceed.