rclone: rclone should continue after "permission denied" (and other) errors
When filing an issue, please include the following information if possible as well as a description of the problem. Make sure you test with the latest beta of rclone.
https://beta.rclone.org/
https://rclone.org/downloads/
If you’ve just got a question or aren’t sure if you’ve found a bug then please use the rclone forum instead of filing an issue.
What is your rclone version (eg output from
rclone -V)
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
A log from the command with the
-vvflag (eg output fromrclone -vv copy /tmp remote:tmp)
Backup is with b2. rclone v1.35-92-g18c75a8β Rclone sync
I noticed that when viewing my logs rclone stops, or at least seems to stop at “permission denied” when trying to back up some folders. I don’t mind that it can’t access these folders, however can’t it just skip them, or am I not being patient enough. This was at about 6 this morning and it’s now 9 something, so 3 hours have passed. I’m not worried as most of my stuff are
backed up, however I did want to bring this to your attention just in case this was an issue. outputsamusic.logaq.zip
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 2
- Comments: 19 (9 by maintainers)
@jwiegley
--exclude .Trasheswon’t exclude the directory - you need--exclude .Trashes/**which means exclude Trashes and everything under, Not just exclude Trashes.The above said, I intend to make rclone continue after that sort of error which is what rsync does, and just report an error at the end.
@ncw sounds good! 😃 any idea when rclone will continue after a permission issue?
I am backing up various large directories, so filtering all files / dirs individually is fairly impractical.
Also if new files with perm issues pop up, this invalidates the whole backup process
It’s not important only in case of access denials, but also in case of any other error. For example, I have a situaton where some objects in Swift container are resulting in 404. They are in object list, but actually not accessible due to the Swift cluster issues.
When enabling --log-level=DEBUG and --dump headers, I’m able to list content of a subdirectory and see first object failing with 404, but that’s it. Can’t get any more 404 objects because of rclone failing and exiting. Excluding first failing object does not also work with command lsd.
It should be possible to do lsd and get list of all objects, including erroneous objects.
I think I got it, I’ll do some more testing when I next run the back up which shoudl be after work today.
Thanks so much.
The error is
Which can be worked around with a filter as @ajkis suggested 😃
If you have having trouble with the filters then post them here and we can help you debug them - they should definitely work with b2.
Can you post a log with
-vvplease?You should use filters and exclude those files/folders since its not safe to skip.
@flerim please make a new issue with a way to reproduce and a full log please - thank you 😃
I just had a look at the code. I think it will report errors, but it should traverse as much of the filesystem as it can - is that what you are seeing?
I’m hitting this problem as well in rclone v1.46, and
--ignore-errorsdoesn’t seem to help.2019/04/12 09:28:19 ERROR : : error reading source directory: failed to read directory "": lstat /mnt/d/pagefile.sys: permission deniedThanks for reminding me about this issue @vkukk - I’ll put it on the milestone.
Btw, if I try to ignore a directory that I don’t have permissions to read (such as
.Trashes), rclone still tries to read it, and still triggers an error as a result. It’s as if it’s trying to determine something about the directory, even though it has been excluded with--exclude .Trashes.In doing further testing the program stil stops in the same place. I did exclude those files in the filter however for some reason g2 doesn’t like it. Here is the link to the files.
https://www.dropbox.com/s/sjga0bdvyfgr3ad/b2 logs.zip?dl=1
I have my filters file in there as well, I really have no clue if that will help, but thought I’d include that anyway just in case I mis read the documentation.
Thanks so much and hope that helps a bit wiht my odd issue.
that sounds good, at least it’s worth a try.
Thanks.
I could but I thought b2 didn’t do exclude? I have a file full of filters but b2 I believe still won’t exclude them. I’ll try again and report back when I try the backup again.