dvc: DVCFileSystem.find very slow with many files

Bug Report

Calling find on DVCFileSystem with a repo of about 300,000 files takes about 70 seconds.

Description

Reproduce

This example retrieves all file paths and hashes in a given dataset directory:

from dvc.api import DVCFileSystem

fs = DVCFileSystem(repo_url, rev=rev)
files = fs.find("/Data", detail=True, dvc_only=True)
remote_files = {}
for file, info in files.items():
    hash = info['dvc_info']['md5']
    remote_files[file] = hash

Given that all this information is stored in a single internal json file, I’d expect this to take as long as downloading that file + a little housekeeping. On a repo with 300,000 files, this takes about 70 seconds.

Expected

This example does the same, but accesses internal datastuctures to make it faster:

from dvc.api import DVCFileSystem
from dvc_data.hashfile.tree import Tree

fs = DVCFileSystem(repo_url, rev=rev)
key = fs._get_key_from_relative("Data")
_, dvc_fs, subkey = fs._get_subrepo_info(key)
entry = dvc_fs.fs.index._trie.get(subkey)
entry.obj = Tree.load(entry.remote, entry.hash_info)

remote_files = {}
for ikey, (_, hash_info) in entry.obj.iteritems():
    file = os.path.join(subkey[0], *ikey)
    hash = hash_info.value
    remote_files[file] = hash

On a repo with 300,000 files, this takes about 10 seconds. Most of that time is spent downloading.

A factor of 7 seems like too much overhead to me. Hopefully, this can be improved.

DVC version: 2.38.1 (conda)
---------------------------------
Platform: Python 3.9.15 on Windows-10-10.0.19045-SP0
Subprojects:
        dvc_data = 0.28.4
        dvc_objects = 0.14.0
        dvc_render = 0.0.15
        dvc_task = 0.1.8
        dvclive = 1.3.0
        scmrepo = 0.1.4
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        ssh (sshfs = 0.0.0)

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 2
  • Comments: 17 (8 by maintainers)

Most upvoted comments

@skshetry Yes, that sounds right. Unfortunately that’s how we organize our data. For each case we have a few files and a folder for each case. Changing that structure isn’t really feasable, it would be easier to change to a different version control.

@shcheklein Thanks for the reply!

Our Use case is similar to this: We try to version-control a folder with a lot of dataset-subfolders. Each subfolder contains images (static), different image masks to different topics (versioned) and versioned metadata like bounding boxes and annotations, which are then used for the training of different AI models / products. In each iteration the label data of a single AI model are improved and a new version is trained.

Maybe similar to real world datasets like:

  1. COCO (Common Objects in Context): COCO is a large-scale object detection, segmentation, and captioning dataset. It includes annotations for object detection (bounding boxes), keypoints (skeletons), and segmentation masks.
  2. Open Images V7 - Description (storage.googleapis.com) Open Images is a dataset of ~9M images annotated with image-level labels, object bounding boxes, object segmentation masks, visual relationships, and localized narratives

Our dataset is just a bit bigger and contains data for different AI models. Not every model needs all of the data, but splitting up the data for different ai models doesn’t make much sense for organizing and improving the data.

I also just tried the self.fs.find("/", detail=True, dvc_only=True) call again on the latest dvc version and after reinstalling Ubuntu on the machine. Unfortunately, it seems like it got much slower. The function takes now about 90 minutes to return the info about 508690 files.

Here is the dvc doctor output:

DVC version: 3.55.2 (conda)
---------------------------
Platform: Python 3.10.14 on Linux-6.8.0-45-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.16.6
        dvc_objects = 5.1.0
        dvc_render = 1.0.2
        dvc_task = 0.4.0
        scmrepo = 3.3.6
Supports:
        http (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.5, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2024.6.0)
Config:
        Global: /home/***@***/.config/dvc
        System: /etc/xdg/dvc
Cache types: reflink, symlink
Cache directory: btrfs on /dev/nvme2n1p2
Caches: local
Remotes: local
Workspace directory: btrfs on /dev/nvme2n1p2
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/751a2518d8177e62110b475a90c0a87a

@grahameth How large are the files in that directory?

Well, it’s about 500000 files. The largest file is 600MB, many are couple MB to 100MB and most are just a few KB. In total, it’s about 900GB. I also have an Ubuntu machine running this repo, and the commands all feel very slow there too. But I haven’t timed it there yet. When I find time on Wednesday next week or so, I will try to make a reproducible repo.

@efiop I see much faster times than what’s reported, so I’m not sure it’s worth a lot of time until we identify what’s causing such slowness for @grahameth specifically.

I just tried this with DVC version 3.6.0, and it seems to have gotten worse: On a repo with 500000 tracked files: DVCFileSystem.find(“/”, …) takes about 170 seconds.

The code below which uses internal datastructures takes about 9 seconds. (The code in the first post no longer works on 3.6.0)

from dvc.api import DVCFileSystem
from dvc_data.hashfile.tree import Tree

fs = DVCFileSystem(repo_url, rev=rev)
remote_files = {}
for dir in fs.find("/", maxdepth=1, detail=False, withdirs=True, dvc_only=True):
    key = fs._get_key_from_relative(dir)
    repo, dvc_fs, subkey = fs._get_subrepo_info(key)
    entry = dvc_fs.fs.index._trie.get(subkey)
    storage = dvc_fs.fs.index.storage_map[key].remote
    obj = Tree.load(storage.odb, entry.hash_info, hash_name=storage.odb.hash_name)

    for ikey, (_, hash_info) in obj.iteritems():
        path = os.path.join(subkey[0], *ikey)
        path = path.replace("\\", "/")
        if path[0] == "/":
            path = path[1:]
        hash = hash_info.value
        remote_files[path] = hash

Here is the output of dvc doctor on the client side:

DVC version: 3.6.0 (conda)
--------------------------
Platform: Python 3.10.12 on Windows-10-10.0.19045-SP0
Subprojects:
        dvc_data = 2.6.0
        dvc_objects = 0.24.1
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.1.0
Supports:
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.7.0)
Config:
        Global: C:\Users\deeplearning\AppData\Local\iterative\dvc
        System: C:\ProgramData\iterative\dvc