dvc: sshfs: SFTP mkdir permission error on push
Bug Report
Issue name
using command “dvc push -v” asyncssh.sftp.SFTPPermissionDenied: Permission denied
Description
I had some problems when using the command “dvc push”. I wanted to add some data to the dataset. I pulled from the server and used “dvc push -v”.
Then, the following error popped up.
2021-12-20 18:47:37,032 ERROR: failed to transfer 'md5: 1f18348f32c9a4694f16426798937ae2' - Permission denied
------------------------------------------------------------
Traceback (most recent call last):
File "/snap/dvc/1320/lib/python3.6/site-packages/dvc/objects/transfer.py", line 25, in wrapper
func(path_info, *args, **kwargs)
File "/snap/dvc/1320/lib/python3.6/site-packages/dvc/objects/transfer.py", line 160, in func
obj.path_info, obj.fs, obj.hash_info, verify=verify, move=move
File "/snap/dvc/1320/lib/python3.6/site-packages/dvc/objects/db/base.py", line 113, in add
self._add_file(fs, path_info, cache_info, hash_info, move=move)
File "/snap/dvc/1320/lib/python3.6/site-packages/dvc/objects/db/base.py", line 88, in _add_file
self.makedirs(to_info.parent)
File "/snap/dvc/1320/lib/python3.6/site-packages/dvc/objects/db/base.py", line 67, in makedirs
self.fs.makedirs(path_info)
File "/snap/dvc/1320/lib/python3.6/site-packages/dvc/fs/fsspec_wrapper.py", line 198, in makedirs
self._with_bucket(path_info), exist_ok=kwargs.pop("exist_ok", True)
File "/snap/dvc/1320/lib/python3.6/site-packages/fsspec/asyn.py", line 91, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/snap/dvc/1320/lib/python3.6/site-packages/fsspec/asyn.py", line 71, in sync
raise return_result
File "/snap/dvc/1320/lib/python3.6/site-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/snap/dvc/1320/lib/python3.6/site-packages/sshfs/utils.py", line 27, in wrapper
return await func(*args, **kwargs)
File "/snap/dvc/1320/lib/python3.6/site-packages/sshfs/spec.py", line 232, in _makedirs
await channel.makedirs(path, exist_ok=exist_ok, attrs=attrs)
File "/snap/dvc/1320/lib/python3.6/site-packages/asyncssh/sftp.py", line 2609, in makedirs
await self.mkdir(curpath, attrs)
File "/snap/dvc/1320/lib/python3.6/site-packages/asyncssh/sftp.py", line 3293, in mkdir
await self._handler.mkdir(path, attrs)
File "/snap/dvc/1320/lib/python3.6/site-packages/asyncssh/sftp.py", line 1507, in mkdir
return await self._make_request(FXP_MKDIR, String(path), attrs.encode())
File "/snap/dvc/1320/lib/python3.6/site-packages/asyncssh/sftp.py", line 1189, in _make_request
result = self._packet_handlers[resptype](self, resp)
File "/snap/dvc/1320/lib/python3.6/site-packages/asyncssh/sftp.py", line 1221, in _process_status
raise _construct_sftp_error(code, reason, lang)
asyncssh.sftp.SFTPPermissionDenied: Permission denied
Reproduce
git clone $url
dvc pull
mkdir test_data
vim test_data/test_file.txt
dvc add test_data
dvc push -v
Expected
I tried ssh and sftp connection which worked well. Besides, I found that if I sampled data from the original dataset and made a copy to a new folder and pushed that folder to the server worked. I checked file permission, group and owner were all the same. Can’t figure it out .
Environment information
My dvc version:
DVC version: 2.7.2 (pip)
---------------------------------
Platform: Python 3.6.9 on Linux-5.4.0-90-generic-x86_64-with-Ubuntu-18.04-bionic
Supports:
http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.5),
ssh (sshfs = 2021.8.1)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda2
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/sda2
Repo: dvc, git
Is there a way to fix it ? thank you !
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (10 by maintainers)
Related: https://github.com/fsspec/sshfs/issues/21
Yes, it does! 🥳
@lefos can you try installing this patch and see if it resolves your issue:
Thanks for looking into it @lefos99, this will need to be addressed upstream in sshfs, will try to get a fix for it out soon
Okay, I found the issue!
In my local script with the sshfs, instead of running:
I deactivate the flag
create_parents
If you have this flag activated, then sshfs tries to create the whole path, and my user (because is no admin) doesn’t have a write access to all parent folders.
In my opinion, this flag should be de-activated by default in DVC.