Bug Report

add: Fails to add file to shared cache on windows drive mount

Description

We have a shared dvc cache on a ZFS file server shared via Samba. On Windows machines, it is mounted as a drive letter. At one point, we could dvc add a file from a windows machine, and dvc would move the file to the cache and replace the local file with a symlink, as expected.

Somewhere between v1.7.2 and v1.11.10, it stopped working. Specifically, this is what appears to happen:

1. The file hash is calculated as expected.
2. The file bytes are copied to the cache under the correct name, but with the random suffix, denoting that the copy is in progress.
3. It never seems to be able to rename the cache file to it’s final file name without the random suffix. No error is emitted, even with -v.
4. No .dvc file is created.
5. It errors out at a point where it’s probably trying to replace the local file with a symlink. Output below.

Note that this happens even if I change the cache.type to copy, even if I run from an Admin-elevated cmd.

Note also that the same operation runs just fine using a Ubuntu machine connecting to the same cache.

Reproduce

mkdir D:\dvcregression
mkdir Z:\dvcregression_cache
cd D:\dvcregression
git init
dvc init
dvc config cache.dir Z:\dvcregression_cache
# copy in file TC3-096.oct to D:\dvcregression
dvc add -v TC3-096.oct

And you get the error pasted below.

2021-01-12 15:44:59,435 DEBUG: Check for update is enabled.
2021-01-12 15:44:59,514 DEBUG: Trying to spawn '['daemon', '-q', 'updater']'
2021-01-12 15:45:03,013 DEBUG: Spawned '['daemon', '-q', 'updater']'
2021-01-12 15:45:03,420 DEBUG: Adding 'TC3-096.oct' to '.gitignore'.
2021-01-12 15:45:03,451 DEBUG: Path 'D:\dvcregression\TC3-096.oct' inode '5598938201872874496'
2021-01-12 15:45:03,451 DEBUG: fetched: []
2021-01-12 15:45:04,107 DEBUG: Path 'TC3-096.oct' inode '5598938201872874496'
2021-01-12 15:45:04,107 DEBUG: fetched: []
2021-01-12 15:45:04,107 DEBUG: {'TC3-096.oct': 'modified'}
2021-01-12 15:45:04,107 DEBUG: Path 'D:\dvcregression\TC3-096.oct' inode '5598938201872874496'
2021-01-12 15:45:04,107 DEBUG: fetched: [('1607391879537605376', '364223321', '9baae98c6a197008759288d2dca21ea7', '1610484304107641088')]
2021-01-12 15:45:04,107 DEBUG: Computed stage: 'TC3-096.oct.dvc' md5: 'None'
2021-01-12 15:45:04,123 DEBUG: Saving 'TC3-096.oct' to 'Z:\dvcregression_cache\9b\aae98c6a197008759288d2dca21ea7'.
2021-01-12 15:45:04,123 DEBUG: cache 'Z:\dvcregression_cache\9b\aae98c6a197008759288d2dca21ea7' expected 'HashInfo(name='md5', value='9baae98c6a197008759288d2dca21ea7', dir_info=None, size=364223321, nfiles=None)' actual 'None'
Adding...
2021-01-12 15:45:05,201 DEBUG: fetched: [(0,)]
2021-01-12 15:45:05,217 ERROR: unexpected error - [WinError 5] Access is denied: 'D:\\dvcregression\\TC3-096.oct'
------------------------------------------------------------
Traceback (most recent call last):
  File "shutil.py", line 566, in move
OSError: [WinError 17] The system cannot move the file to a different disk drive: 'D:\\dvcregression\\TC3-096.oct' -> 'Z:\\dvcregression_cache\\9b\\aae98c6a197008759288d2dca21ea7.mFkbHoodLwcYscY9yK6gCZ'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dvc\main.py", line 90, in main
  File "dvc\command\add.py", line 24, in run
  File "dvc\repo\__init__.py", line 54, in wrapper
  File "dvc\repo\scm_context.py", line 4, in run
  File "dvc\repo\add.py", line 85, in add
  File "dvc\repo\add.py", line 124, in _process_stages
  File "site-packages\funcy\decorators.py", line 39, in wrapper
  File "dvc\stage\decorators.py", line 36, in rwlocked
  File "site-packages\funcy\decorators.py", line 60, in __call__
  File "dvc\stage\__init__.py", line 478, in commit
  File "dvc\output\base.py", line 297, in commit
  File "site-packages\funcy\decorators.py", line 39, in wrapper
  File "dvc\cache\base.py", line 40, in use_state
  File "site-packages\funcy\decorators.py", line 60, in __call__
  File "dvc\cache\base.py", line 317, in save
  File "dvc\cache\base.py", line 326, in _save
  File "dvc\cache\base.py", line 194, in _save_file
  File "dvc\tree\local.py", line 177, in move
  File "dvc\utils\fs.py", line 111, in move
  File "shutil.py", line 581, in move
PermissionError: [WinError 5] Access is denied: 'D:\\dvcregression\\TC3-096.oct'
------------------------------------------------------------
2021-01-12 15:45:05,795 DEBUG: Version info for developers:
DVC version: 1.11.10 (exe)
---------------------------------
Platform: Python 3.7.9 on Windows-10-10.0.18362-SP0
Supports: All remotes
Cache types: symlink
Cache directory: NTFS on Z:\
Caches: local
Remotes: None
Workspace directory: NTFS on D:\
Repo: dvc, git

Expected

Expect that dvc would continue to support a shared cache like this.

Environment information

Output of dvc version:

DVC version: 1.11.10 (exe)
---------------------------------
Platform: Python 3.7.9 on Windows-10-10.0.18362-SP0
Supports: All remotes
Cache types: symlink
Cache directory: NTFS on Z:\
Caches: local
Remotes: None
Workspace directory: NTFS on D:\
Repo: dvc, git

Additional Information (if any):

From the error output, it seems that dvc expects that a “local” cache is always going to be on the same drive, which is not the case here. Is there a way to tell it not to have that expectation?

I tried reverting to 1.7.2 and confirmed it still works there. I can try searching for the breaking version, but I’d appreciate some guidance.