dvc: Unexpected error when cache and artifacts are not on the same partition on SSH

Setup

pip installed, version 0.66.1. Platform: Ubuntu 18.04

Steps to reproduce

  1. Create SSH remote cache, and configure it as SSH cache.
  2. Create directory on the same computer, but on other partition. Add some test.txt to this dir.
  3. Add this remote directory as SSH remote called workspace.
  4. Run dvc add remote://workspace/test.txt -f test.dvc.

Bash script reproducing error:

# arrange
workspace=$(mktemp -d)
cache=/tmp/ram

mkdir -p /tmp/ram
sudo mount -t tmpfs -o size=512M tmpfs /tmp/ram  # creating distinct partition for cache

cd $(mktemp -d)

dvc init --no-scm

dvc remote add workspace ssh://localhost/${workspace}
dvc remote add cache ssh://localhost/${cache}
dvc config cache.ssh cache

echo "foo" > ${workspace}/foo

# act
dvc add remote://workspace/foo

# cleanup
sudo umount /tmp/ram
rmdir /tmp/ram

Observed behavior

100%|██████████|Add                                                                                          1/1 [00:00<00:00,  1.02file/s]
ERROR: unexpected error - Failure

Expected behavior

I believe that by design DVC should raise error in this case - in order to avoid implicit movement of large files between partitions.

However, the error message should suggest solution:

ERROR: File move to cache failed.
Ensure that file remote://workspace/test.txt is on the same partition as SSH cache.

Some people might by mistake put file to different partition (it happened to me! - such message could save me a bit of time 😉).


Because I had already debugged/studied --verbose DVC to understand the source of the problem, I could try to implement such message through Pull Request.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 15 (6 by maintainers)

Most upvoted comments

@shcheklein , yes 😃