dvc: `poetry add dvc[s3]` does not install `boto3`

Bug Report

Description

Installing dvc[s3] with poetry does not install boto3, so dvc push or dvc pull do not work.

Reproduce

poetry init -q
poetry add dvc[s3]
poetry run python3 -c "import boto3"

Expected

It is expected that boto3 is installed.

Environment information

Python 3.8, poetry 1.1.12, DVC 2.9.3.

Output of dvc doctor:

DVC version: 2.9.3 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.10.60.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
Supports:
        webhdfs (fsspec = 2021.11.1),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)```

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 3
Comments: 16 (8 by maintainers)

Commits related to this issue

s3: add aiobotocore as dependency explicitly Fixes #7257 and fixes #7475. — committed to skshetry/dvc by skshetry 2 years ago
s3: add aiobotocore as dependency explicitly Fixes #7257 and fixes #7475. — committed to iterative/dvc by skshetry 2 years ago

Most upvoted comments

@dtrifiro That one should be solved by pip install -U pip, there was some bug in dep resolution in pip that got fixed.

efiop on Jan 12, 2022

dvc==2.10.0 is released which should fix the issue. Let us know if there’s any issue.

skshetry on Mar 30, 2022

@tomliptrot explicitly specifying boto3 = "1.19.8" in pyproject.toml functioned for me as a workaround.

ikrets on Jan 13, 2022

I get a similar issue when trying to install boto3 after installing dvc[s3].

poetry init -n
poetry add "dvc[s3]"
poetry add boto3

SolverProblemError

  Because s3fs (2022.1.0) depends on aiobotocore (>=2.1.0,<2.2.0)
   and s3fs (2021.11.1) depends on aiobotocore (>=2.0.1,<2.1.0), s3fs (2021.11.1 || 2022.1.0) requires aiobotocore (>=2.0.1,<2.2.0).
  Because no versions of aiobotocore match >2.1.0,<2.2.0
   and aiobotocore (2.1.0) depends on botocore (>=1.23.24,<1.23.25), aiobotocore (>=2.1.0,<2.2.0) requires botocore (>=1.23.24,<1.23.25).
  Thus, s3fs (2021.11.1 || 2022.1.0) requires botocore (>=1.23.24,<1.23.25) or aiobotocore (>=2.0.1,<2.1.0).
  Because aiobotocore (2.0.1) depends on botocore (>=1.22.8,<1.22.9)
   and no versions of aiobotocore match >2.0.1,<2.1.0, aiobotocore (>=2.0.1,<2.1.0) requires botocore (>=1.22.8,<1.22.9).
  Thus, s3fs (2021.11.1 || 2022.1.0) requires botocore (>=1.22.8,<1.22.9 || >=1.23.24,<1.23.25).
  Because no versions of boto3 match >1.20.34,<2.0.0
   and boto3 (1.20.34) depends on botocore (>=1.23.34,<1.24.0), boto3 (>=1.20.34,<2.0.0) requires botocore (>=1.23.34,<1.24.0).
  Thus, boto3 (>=1.20.34,<2.0.0) is incompatible with s3fs (2021.11.1 || 2022.1.0).
  And because no versions of s3fs match >2021.11.1,<2022.1.0 || >2022.1.0, boto3 (>=1.20.34,<2.0.0) is incompatible with s3fs (>=2021.11.1).
  And because dvc (2.9.3) depends on s3fs (>=2021.11.1)
   and no versions of dvc match >2.9.3,<3.0.0, boto3 (>=1.20.34,<2.0.0) is incompatible with dvc (>=2.9.3,<3.0.0).
  So, because dvc-copat-test depends on both dvc (^2.9.3) and boto3 (^1.20.34), version solving failed.

My investigations led to this issue https://github.com/fsspec/s3fs/issues/357

I tried installing just dvc and adding s3fs<0.5.0 separately.

poetry add dvc
poetry add s3fs"<0.5.0"
poetry add boto3

This allowed me to install boto3, but now dvc does not work.

 ➜ dvc push
ERROR: unexpected error - __init__() got an unexpected keyword argument 'cache_regions'

tomliptrot on Jan 13, 2022

This looks like a bug in poetry, similar to https://github.com/iterative/dvc/issues/6886. See https://github.com/python-poetry/poetry/issues/4683.

A workaround may be to just explicitly specify dependencies in dvc as we did with others.

skshetry on Jan 12, 2022