dvc: Permission denied with windows share accessed from a linux dvc client

Bug Report

Please provide information about your setup

Remote is set to a network drive on a windows share. Reading/Writing to this share is possible.

Output of dvc version:

$ dvc version

DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.6.8 on Linux-4.18.0-147.3.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Supports: http, https
Cache types: reflink, hardlink, symlink
Repo: dvc, git

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[user@localhost <directory>] $ dvc -v push
ERROR: failed to upload '.dvc/cache/93/ce4560ea6eeb07f6c5e5b45542512a' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/93/ce4560ea6eeb07f6c5e5b45542512a' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/93/ce4560ea6eeb07f6c5e5b45542512a.eePk9P9PgCP958UeNydbLp.tmp'
ERROR: failed to upload '.dvc/cache/96/0c155eac0009d3b6dbc019a84f1b6c.dir' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/96/0c155eac0009d3b6dbc019a84f1b6c.dir'
ERROR: failed to upload '.dvc/cache/58/870323aea899a9a42397776edbced7' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/58/870323aea899a9a42397776edbced7' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/58/870323aea899a9a42397776edbced7.mA8BsnQLEBGokvS3k7WuqD.tmp'
ERROR: failed to upload '.dvc/cache/ba/9a3ef332715fc48c8928ff317a835b' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-remote/ba/9a3ef332715fc48c8928ff317a835b' - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-remote/ba/9a3ef332715fc48c8928ff317a835b.DWsNop3is3D7M2Dnwzd34j.tmp'
ERROR: failed to push data to the cloud - 4 files failed to upload`

The files do appear in the remote directory

If the cache gets set onto the same windows fileserver the error is more verbose.

$ dvc add -v unittests/integration_test/testdata/
2020-10-29 07:44:56,040 DEBUG: Check for update is enabled.
2020-10-29 07:44:56,043 DEBUG: fetched: [(3,)]
2020-10-29 07:44:57,640 DEBUG: Adding 'unittests/integration_test/testdata' to 'unittests/integration_test/.gitignore'.
2020-10-29 07:44:57,641 DEBUG: Path '<XY>/unittests/integration_test/testdata' inode '403027213'
2020-10-29 07:44:57,642 DEBUG: fetched: []
2020-10-29 07:44:57,644 DEBUG: Path '<XY>/unittests/integration_test/testdata/testfile.slow' inode '403027257'
2020-10-29 07:44:57,644 DEBUG: fetched: []
2020-10-29 07:44:57,644 DEBUG: Path '<XY>/unittests/integration_test/testdata/testfile.fast' inode '403027259'
2020-10-29 07:44:57,645 DEBUG: fetched: []
2020-10-29 07:44:57,645 DEBUG: Path '<XY>/unittests/integration_test/testdata/.gitignore' inode '403027228'
2020-10-29 07:44:57,645 DEBUG: fetched: []
2020-10-29 07:44:59,373 DEBUG: Uploading '../../../../../tmp/tmp7txde11j' to '../../../../../mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp'
Adding...
2020-10-29 07:44:59,387 DEBUG: fetched: [(0,)]
2020-10-29 07:44:59,389 ERROR: unexpected error - [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp.3QNBXQxKiMMXQ9Puv3DkWy.tmp'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/dvc/main.py", line 76, in main
    ret = cmd.run()
  File "/usr/local/lib/python3.6/site-packages/dvc/command/add.py", line 22, in run
    external=self.args.external,
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/__init__.py", line 54, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/dvc/repo/add.py", line 90, in add
    stage.save()
  File "/usr/local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 386, in save
    self.save_outs(allow_missing=allow_missing)
  File "/usr/local/lib/python3.6/site-packages/dvc/stage/__init__.py", line 398, in save_outs
    out.save()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 278, in save
    if not self.changed():
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 220, in changed
    status = self.status()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 217, in status
    return self.workspace_status()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 205, in workspace_status
    if self.changed_checksum():
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 191, in changed_checksum
    return self.hash_info != self.get_hash()
  File "/usr/local/lib/python3.6/site-packages/dvc/output/base.py", line 180, in get_hash
    return self.cache.get_hash(self.tree, self.path_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 50, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 709, in get_hash
    hash_info = tree.get_hash(path_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 45, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 271, in get_hash
    hash_info = self.get_dir_hash(path_info, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 45, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 340, in get_dir_hash
    return self.repo.cache.local.save_dir_info(dir_info)
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 50, in use_state
    return call()
  File "/usr/local/lib/python3.6/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 301, in save_dir_info
    hi, tmp_info = self._get_dir_info_hash(dir_info)
  File "/usr/local/lib/python3.6/site-packages/dvc/cache/base.py", line 284, in _get_dir_info_hash
    self.tree.upload(from_info, to_info, no_progress_bar=True)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/base.py", line 360, in upload
    no_progress_bar=no_progress_bar,
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/local.py", line 331, in _upload
    self.protect(tmp_file)
  File "/usr/local/lib/python3.6/site-packages/dvc/tree/local.py", line 291, in protect
    os.chmod(path, mode)
PermissionError: [Errno 1] Operation not permitted: '/mnt/networkdrives/<XXX>/DVC/dvc-cache/.59MZDNHC55i69CxyW2HX7b.tmp.3QNBXQxKiMMXQ9Puv3DkWy.tmp'
------------------------------------------------------------
2020-10-29 07:44:59,416 DEBUG: Version info for developers:
DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.6.8 on Linux-4.18.0-147.3.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
Supports: http, https
Cache types: symlink
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-10-29 07:44:59,418 DEBUG: Analytics is enabled.
2020-10-29 07:44:59,727 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpoqoq9g77']'
2020-10-29 07:44:59,728 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpoqoq9g77']'

Also in this case the files do appear on the remote filesystem.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (10 by maintainers)

Most upvoted comments

Closing as stale, @gillhofer feel free to reopen if the problem still occurs.

pared on Dec 18, 2020

@shcheklein I would not say that we don’t support CIFS. Only thing that is not working is a optimization we implemented in https://github.com/iterative/dvc/pull/3472, but it does not prevent DVC from working properly. It just makes it work slower.

@gillhofer That raises new question: whether we should have some kind of mechanism allowing user to say “I know what I am doing, trust the remote/cache even if its unprotected”. It sounds like it could be useful, in some scenarios, especially CI, where you not necessarily edit anything, but rather check out and run.

pared on Nov 18, 2020

@pared Ok, that explains the behaviour.

However, with this setup there seems to be a general issue when CIFS mounts are used as cache, as CIFS does not deal at all with file permissions. From the CIFS Documentation.

The core CIFS protocol does not provide unix ownership information or mode for files and directories. Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. Attempting to change these values via chmod/chown will return success but have no effect.

This explains why your code runs fine, but has no effect on the file permissions.

If I understand this right, than either some config options is necessary to prevent md5 calculation on CIFS mounts or using something other than CIFS for mounting.

If this is right, than CIFS is not really practical in our case for having access to a shared cache, which brings us back to the start.

gillhofer on Nov 17, 2020

I’m assuming this network share is mounted via SMB/CIFS?

You most likely need to adjust your fstab mount settings in order for chmod to work correctly. IIRC it will likely require either mounting the share with noperm or with uid and gid set to the user and group ID values for the user running DVC.

pmrowla on Oct 29, 2020