Finally think I found a way to reproduce this (or something similar) internally. @iterative/dvc You need to configure access to the aws sandbox and then run this script:

set -x
rm -rf working not-working
echo "Pull succeeds when correct profile is set"
git clone https://github.com/dberenbaum/download-dvc-dir working
cd working
AWS_PROFILE=iterative-sandbox dvc pull
git clone https://github.com/dberenbaum/download-dvc-dir not-working
cd not-working
echo "Pull fails when invalid profile is set"
AWS_PROFILE=invalid dvc pull
echo "Pull still fails after correcting profile"
AWS_PROFILE=iterative-sandbox dvc pull

Here’s the output I see:

+ rm -rf working not-working
+ echo 'Pull succeeds when correct profile is set'
Pull succeeds when correct profile is set
+ git clone https://github.com/dberenbaum/download-dvc-dir working
Cloning into 'working'...
remote: Enumerating objects: 210, done.
remote: Counting objects: 100% (210/210), done.
remote: Compressing objects: 100% (205/205), done.
remote: Total 210 (delta 4), reused 208 (delta 2), pack-reused 0
Receiving objects: 100% (210/210), 18.04 MiB | 5.31 MiB/s, done.
Resolving deltas: 100% (4/4), done.
+ cd working
+ AWS_PROFILE=iterative-sandbox
+ dvc pull
Collecting                                                      |189 [00:07, 24.1entry/s]
Fetching
Building workspace index                                       |187 [00:00, 1.87kentry/s]
Comparing indexes                                              |190 [00:00, 24.9kentry/s]
Applying changes                                               |2.00 [00:00,   486file/s]
A       model.pkl
A       lstm/
2 files added and 186 files fetched
+ git clone https://github.com/dberenbaum/download-dvc-dir not-working
Cloning into 'not-working'...
remote: Enumerating objects: 210, done.
remote: Counting objects: 100% (210/210), done.
remote: Compressing objects: 100% (205/205), done.
remote: Total 210 (delta 4), reused 208 (delta 2), pack-reused 0
Receiving objects: 100% (210/210), 18.04 MiB | 3.17 MiB/s, done.
Resolving deltas: 100% (4/4), done.
+ cd not-working
+ echo 'Pull fails when invalid profile is set'
Pull fails when invalid profile is set
+ AWS_PROFILE=invalid
+ dvc pull
Collecting                                                     |3.00 [00:06, 2.06s/entry]
ERROR: failed to connect to s3 (dave-sandbox/cache) - The config profile (invalid) could not be found
ERROR: failed to connect to s3 (dave-sandbox/cache/files/md5) - The config profile (invalid) could not be found
Fetching
ERROR: failed to pull data from the cloud - 2 files failed to download
+ echo 'Pull still fails after correcting profile'
Pull still fails after correcting profile
+ AWS_PROFILE=iterative-sandbox
+ dvc pull
Collecting                                                     |0.00 [00:06,    ?entry/s]
Fetching
Building workspace index                                       |187 [00:00, 1.60kentry/s]
Comparing indexes                                               |190 [00:03, 59.7entry/s]
WARNING: No file hash info found for '/private/tmp/working/not-working/lstm/epoch=0-step=2.ckpt'. It won't be created.
Applying changes                                               |1.00 [00:00,   127file/s]
A       model.pkl
1 file added and 2 files fetched
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
lstm
Is your cache up to date?
<https://error.dvc.org/missing-files>

Works for both external and internal caches, also on the newly released 3.14. Thanks a lot !

Can I remove all the directories inside /var/tmp/dvc/repo ? What are they for ? If they can introduce hard to debug bugs like this one, should they be perhaps nuked when you install a new version, although I am not sure how that could help. Or at least there should be docs somewhere to suggest removing this directory.

On another note, I’ve been using DVC for a few months now, and my initial impression is that new releases are happening perhaps too frequently and often breaking core functionality. Don’t get me wrong, but shouldn’t there be more focus on stability ? I already received some feedback from colleagues that get quite hesitant about using DVC at all, when they see these types of bugs.

@dberenbaum As promised, I’ll look into dropping fetch cache in drop_data_index and will close it.

EDIT: doesn’t seem like we need to for now. Closing.

Ok, so there were a few things here, but the main one is really https://github.com/iterative/dvc-data/pull/447/commits/99fd441e2f3fa69939a25d00eb5714e640898fe1 , where we were thinking that directory entries were supposed to be loaded later, but they were not (e.g. because of missing objects or missing creds or auth error or something else), and that was then cached in site_cache_dir and was not affected by drop_data_index because it is not under tree prefix (which it probably should drop, taking a look).

After https://github.com/iterative/dvc-data/pull/447 , those repro scripts from above start to work for me just fine. Will take a final look in the morning.

@dberenbaum Thanks again for the repro script, that helps a lot! 🙏

Here is a simplified local version that shows the same problem and narrows it down:

#!/bin/bash
set -e
set -x

rm -rf mytest
mkdir mytest
cd mytest

mkdir mydatareg
pushd mydatareg
MYDATAREG=$(pwd)
git init
dvc init
git add .
git commit -m "init"
mkdir data
echo foo > data/foo
dvc add data
git add data.dvc .gitignore
git commit -m "data"
popd

mkdir myrepo
cd myrepo
dvc init --no-scm
dvc import $MYDATAREG data

rm -rf .dvc/cache
rm -rf /Library/Caches/dvc
rm -rf data

mv $MYDATAREG/.dvc/cache $MYDATAREG/.dvc/cache.back

dvc pull || echo FAILED

mv $MYDATAREG/.dvc/cache.back $MYDATAREG/.dvc/cache

dvc pull || echo STILL FAILED

rm -rf /Library/Caches/dvc  # obviously this is only relevant for macs

dvc pull

So it is index for import that is not reloaded for the second pull after the first one failed. Looking into it…

Nice, thank you for the repro script! It looks like a similar problem to one that was recently raised here. The whole thread is long and not all related, but that comment looks similar to what you raise.

@efiop WDYT? Below I slightly adjust the script to do dvc import --no-download followed by dvc pull, which also unexpectedly fails.

FROM ubuntu:20.04
# Deps
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
        sudo git python3.8-dev python3-pip \
    && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir dvc==3.19.0
# Create a non-root user (perhaps not required but is closer to my setup)
ARG user=user
ARG uid=1000
ARG gid=${uid}
RUN groupadd --gid ${gid} ${user} \
    && useradd -s /bin/bash --uid ${uid} --gid ${gid} -m ${user} \
    && echo ${user} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${user} \
    && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \
    && chmod 0440 /etc/sudoers.d/${user}
USER ${user}
# Configure git
RUN git config --global user.name ${user} \
    && git config --global user.email ${user}@example.com
# DVC data reg git repo with a directory as remote storage
RUN mkdir /home/${user}/storage && mkdir /home/${user}/data_reg
WORKDIR /home/${user}/data_reg
RUN git init && dvc init
RUN dvc config core.autostage true
RUN dvc remote add -d storage /home/${user}/storage
RUN mkdir -p data/raw && echo 1 > data/raw/data.txt
RUN dvc add data/raw && dvc push
RUN git status && git commit -m "add test data"
# clean all local data
# RUN rm -rf /var/tmp/dvc \
#     && rm -rf .dvc/cache \
#     && rm -rf .dvc/tmp \
#     && rm -rf data/raw
# DVC model training repo, imports data from data_reg git repo
RUN mkdir -p /home/${user}/model_repo/data
WORKDIR /home/${user}/model_repo
RUN dvc init --no-scm
# This part runs without issue
RUN dvc import --no-download ../data_reg data/raw -o data/raw
RUN cat data/raw.dvc
# This is where it fails...
RUN dvc pull data/raw.dvc --verbose

I tried my hand at a minimum reproducible error with a Dockerfile. It does produce an error but it is not the same as the original post. Let me know if I am missing anything or should move to a new issue:

FROM ubuntu:20.04
# Deps
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
        sudo git python3.8-dev python3-pip \
    && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir dvc==3.19.0
# Create a non-root user (perhaps not required but is closer to my setup)
ARG user=user
ARG uid=1000
ARG gid=${uid}
RUN groupadd --gid ${gid} ${user} \
    && useradd -s /bin/bash --uid ${uid} --gid ${gid} -m ${user} \
    && echo ${user} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${user} \
    && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \
    && chmod 0440 /etc/sudoers.d/${user}
USER ${user}
# Configure git
RUN git config --global user.name ${user} \
    && git config --global user.email ${user}@example.com
# DVC data reg git repo with a directory as remote storage
RUN mkdir /home/${user}/storage && mkdir /home/${user}/data_reg
WORKDIR /home/${user}/data_reg
RUN git init && dvc init
RUN dvc config core.autostage true
RUN dvc remote add -d storage /home/${user}/storage
RUN mkdir -p data/raw && echo 1 > data/raw/data.txt
RUN dvc add data/raw && dvc push
RUN git status && git commit -m "add test data"
# clean all local data
RUN rm -rf /var/tmp/dvc \
    && rm -rf .dvc/cache \
    && rm -rf .dvc/tmp \
    && rm -rf data/raw
# DVC model training repo, imports data from data_reg git repo
RUN mkdir -p /home/${user}/model_repo/data
WORKDIR /home/${user}/model_repo
RUN dvc init --no-scm
# This part downloads the content without issue
RUN dvc import ../data_reg data/raw -o data/raw
RUN cat data/raw.dvc
RUN ls -l data/raw
# Manually remove cache to illustrate.
# Same result as if we ran `dvc import --no-download`
RUN rm -rf /var/tmp/dvc \
    && rm -rf .dvc/cache \
    && rm -rf data/raw
# This is where it fails...
RUN dvc pull data/raw.dvc --verbose

Running docker build --progress=plain . gives me the following output:

...
#25 0.990 2023-09-11 19:18:47,784 DEBUG: failed to load ('data', 'raw') from storage local (data/raw) - failed to load directory ('data', 'raw'): [Errno 2] No such file or directory: '/home/user/model_repo/.dvc/cache/files/md5/54/3ef358e20c4e74b80d2c71243210f0.dir'
#25 0.990 Traceback (most recent call last):
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 541, in _load_from_storage
#25 0.990     _load_from_object_storage(trie, entry, storage)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 477, in _load_from_object_storage
#25 0.990     obj = Tree.load(storage.odb, root_entry.hash_info, hash_name=storage.odb.hash_name)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/hashfile/tree.py", line 193, in load
#25 0.990     with obj.fs.open(obj.path, "r") as fobj:
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_objects/fs/base.py", line 228, in open
#25 0.990     return self.fs.open(path, mode=mode, **kwargs)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_objects/fs/local.py", line 136, in open
#25 0.990     return open(path, mode=mode, encoding=encoding)
#25 0.990 FileNotFoundError: [Errno 2] No such file or directory: '/home/user/model_repo/.dvc/cache/files/md5/54/3ef358e20c4e74b80d2c71243210f0.dir'
#25 0.990 
#25 0.990 The above exception was the direct cause of the following exception:
#25 0.990 
#25 0.990 Traceback (most recent call last):
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 543, in _load_from_storage
#25 0.990     _load_from_file_storage(trie, entry, storage)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 515, in _load_from_file_storage
#25 0.990     for entry in build_entries(path, fs):
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/build.py", line 54, in build_entries
#25 0.990     for root, dirs, files in walk_iter:
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/fsspec/spec.py", line 418, in walk
#25 0.990     listing = self.ls(path, detail=True, **kwargs)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc/fs/dvc.py", line 299, in ls
#25 0.990     for info in dvc_fs.ls(dvc_path, detail=True):
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_objects/fs/base.py", line 425, in ls
#25 0.990     return self.fs.ls(path, detail=detail, **kwargs)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/fs.py", line 134, in ls
#25 0.990     for key, info in self.index.ls(root_key, detail=True):
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 693, in ls
#25 0.990     self._ensure_loaded(root_key)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 690, in _ensure_loaded
#25 0.990     self._load(prefix, entry)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 636, in _load
#25 0.990     self.onerror(entry, exc)
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 568, in _onerror
#25 0.990     raise exc
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 634, in _load
#25 0.990     _load_from_storage(self._trie, entry, self.storage_map[key])
#25 0.990   File "/usr/local/lib/python3.8/dist-packages/dvc_data/index/index.py", line 556, in _load_from_storage
#25 0.990     raise DataIndexDirError(f"failed to load directory {entry.key}") from last_exc
#25 0.990 dvc_data.index.index.DataIndexDirError: failed to load directory ('data', 'raw')
#25 0.990 
#25 0.991 2023-09-11 19:18:47,785 DEBUG: failed to create '/home/user/model_repo/data/raw' from 'None'
#25 0.991 NoneType: None
#25 0.991 
#25 0.991 Everything is up to date.
#25 0.991 2023-09-11 19:18:47,785 ERROR: failed to pull data from the cloud - Checkout failed for following targets:
#25 0.991 data/raw
#25 0.991 Is your cache up to date?
#25 0.991 <https://error.dvc.org/missing-files>
#25 0.991 Traceback (most recent call last):
#25 0.991   File "/usr/local/lib/python3.8/dist-packages/dvc/commands/data_sync.py", line 31, in run
#25 0.991     stats = self.repo.pull(
#25 0.991   File "/usr/local/lib/python3.8/dist-packages/dvc/repo/__init__.py", line 61, in wrapper
#25 0.991     return f(repo, *args, **kwargs)
#25 0.991   File "/usr/local/lib/python3.8/dist-packages/dvc/repo/pull.py", line 43, in pull
#25 0.991     stats = self.checkout(
#25 0.991   File "/usr/local/lib/python3.8/dist-packages/dvc/repo/__init__.py", line 61, in wrapper
#25 0.991     return f(repo, *args, **kwargs)
#25 0.991   File "/usr/local/lib/python3.8/dist-packages/dvc/repo/checkout.py", line 204, in checkout
#25 0.991     raise CheckoutError([relpath(out_path) for out_path in failed], stats)
#25 0.991 dvc.exceptions.CheckoutError: Checkout failed for following targets:
#25 0.991 data/raw
#25 0.991 Is your cache up to date?
#25 0.991 <https://error.dvc.org/missing-files>
...