dvc: add: start having ERROR: unexpected error - 'Tqdm' object has no attribute 'desc' when upgrade from 2.10.2 to 2.11.0

Bug Report

Description

Running dvc add ./data shows error ERROR: unexpected error - 'Tqdm' object has no attribute 'desc' on version 2.11.0. In my case, one interesting thing is that if the data folder is generated using dev data with the folder size ~55MB, both 2.10.2 and 2.11.0 work, but if I use prod data with the folder size ~2.59GB, only 2.10.2 works.

Reproduce

dvc remote default prod
dvc add ./data

Expected

2.11.0 should work on both dev and prod data like how 2.10.2 does. It should not depend on the data size.

Environment information

dvc doctor
DVC version: 2.11.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.10.104-linuxkit-x86_64-with-glibc2.29
Supports:
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.5.0, boto3 = 1.21.21)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 16 (13 by maintainers)

Commits related to this issue

Most upvoted comments

OK, guys, I’ve found the issue! See this chunk of code in tqdm. When being run in a non-interactive environment (isatty() is False), the tqdm constructor hits this early return and doesn’t set the desc property.

So, should I submit a patch to tqdm or should dvc_objects stop relying on existence of the desc attribute?

Update I also opened https://github.com/iterative/dvc-objects/pull/92. Let’s solve this!

Ah srry for not seeing this earlier.

Thanks @sjawhar & @daavoo for the debugging efforts!

A few points:

Or it’s just about size of data you add?

It’s about the size of the file, the line raising the error comes from:

https://github.com/iterative/dvc-objects/blob/b469fb63c4479c85f064c33696ef4f18ed92f65c/src/dvc_objects/hash.py#L133-L155

@casperdcl Maybe you have some idea here?