dvc: DVCFileSystem.find very slow with many files
Bug Report
Calling find on DVCFileSystem with a repo of about 300,000 files takes about 70 seconds.
Description
Reproduce
This example retrieves all file paths and hashes in a given dataset directory:
from dvc.api import DVCFileSystem
fs = DVCFileSystem(repo_url, rev=rev)
files = fs.find("/Data", detail=True, dvc_only=True)
remote_files = {}
for file, info in files.items():
hash = info['dvc_info']['md5']
remote_files[file] = hash
Given that all this information is stored in a single internal json file, I’d expect this to take as long as downloading that file + a little housekeeping. On a repo with 300,000 files, this takes about 70 seconds.
Expected
This example does the same, but accesses internal datastuctures to make it faster:
from dvc.api import DVCFileSystem
from dvc_data.hashfile.tree import Tree
fs = DVCFileSystem(repo_url, rev=rev)
key = fs._get_key_from_relative("Data")
_, dvc_fs, subkey = fs._get_subrepo_info(key)
entry = dvc_fs.fs.index._trie.get(subkey)
entry.obj = Tree.load(entry.remote, entry.hash_info)
remote_files = {}
for ikey, (_, hash_info) in entry.obj.iteritems():
file = os.path.join(subkey[0], *ikey)
hash = hash_info.value
remote_files[file] = hash
On a repo with 300,000 files, this takes about 10 seconds. Most of that time is spent downloading.
A factor of 7 seems like too much overhead to me. Hopefully, this can be improved.
DVC version: 2.38.1 (conda)
---------------------------------
Platform: Python 3.9.15 on Windows-10-10.0.19045-SP0
Subprojects:
dvc_data = 0.28.4
dvc_objects = 0.14.0
dvc_render = 0.0.15
dvc_task = 0.1.8
dvclive = 1.3.0
scmrepo = 0.1.4
Supports:
http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
ssh (sshfs = 0.0.0)
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 2
- Comments: 17 (8 by maintainers)
@skshetry Yes, that sounds right. Unfortunately that’s how we organize our data. For each case we have a few files and a folder for each case. Changing that structure isn’t really feasable, it would be easier to change to a different version control.
@shcheklein Thanks for the reply!
Our Use case is similar to this: We try to version-control a folder with a lot of dataset-subfolders. Each subfolder contains images (static), different image masks to different topics (versioned) and versioned metadata like bounding boxes and annotations, which are then used for the training of different AI models / products. In each iteration the label data of a single AI model are improved and a new version is trained.
Maybe similar to real world datasets like:
Our dataset is just a bit bigger and contains data for different AI models. Not every model needs all of the data, but splitting up the data for different ai models doesn’t make much sense for organizing and improving the data.
I also just tried the
self.fs.find("/", detail=True, dvc_only=True)call again on the latest dvc version and after reinstalling Ubuntu on the machine. Unfortunately, it seems like it got much slower. The function takes now about 90 minutes to return the info about 508690 files.Here is the dvc doctor output:
Well, it’s about 500000 files. The largest file is 600MB, many are couple MB to 100MB and most are just a few KB. In total, it’s about 900GB. I also have an Ubuntu machine running this repo, and the commands all feel very slow there too. But I haven’t timed it there yet. When I find time on Wednesday next week or so, I will try to make a reproducible repo.
@efiop I see much faster times than what’s reported, so I’m not sure it’s worth a lot of time until we identify what’s causing such slowness for @grahameth specifically.
I just tried this with DVC version 3.6.0, and it seems to have gotten worse: On a repo with 500000 tracked files: DVCFileSystem.find(“/”, …) takes about 170 seconds.
The code below which uses internal datastructures takes about 9 seconds. (The code in the first post no longer works on 3.6.0)
Here is the output of dvc doctor on the client side: