rclone: rclone hanging after retries exceed for a file

I searched other issues but I wasn’t 100% sure if they were similar enough to bump them. So here is a new one that can be tied together.

I have two mounts. 1 GDrive and 1 ACD. I push the same content with rclone itselv to both via mounts. I get lots of errors on both. The errors are more plentiful than the successes. 😃

So the path looks like this:

local volume-> rclone copy /data/Media1 /data/Media3 where Media1 is local content and Media3 is a gdrive crypt mount. local volume-> rclone copy /data/Media1 /data/Media2 where Media1 is local content and Media2 is a ACD crypt mount.

This is an example of the log output from ACD:

2017/03/06 07:30:02 INFO  : Videos/Movies/Movie1/Movie1.nfo: Copied (new)
2017/03/06 07:30:03 INFO  : Videos/Series/Series2/Season.1/.actors/Jason_Watkins.jpg: Copied (new)
2017/03/06 07:30:07 NOTICE: Videos/Series/Series3/Season.2/Series3.S02E03.720p.HDTV.x264-AVS[PRiME]/Torrent Downloaded From 1337x.to.txt: Removing partially written file on error: close /data/Media2/Videos/Series/Series3/Season.2/Series3.S02E03.720p.HDTV.x264-AVS[PRiME]/Torrent Downloaded From 1337x.to.txt: input/output error
2017/03/06 07:30:07 ERROR : Videos/Series/Series3/Season.2/Series3.S02E03.720p.HDTV.x264-AVS[PRiME]/Torrent Downloaded From 1337x.to.txt: Failed to remove partially written file: remove /data/Media2/Videos/Series/Series3/Season.2/Series3.S02E03.720p.HDTV.x264-AVS[PRiME]/Torrent Downloaded From 1337x.to.txt: no such file or directory
2017/03/06 07:30:07 ERROR : Videos/Series/Series3/Season.2/Series3.S02E03.720p.HDTV.x264-AVS[PRiME]/Torrent Downloaded From 1337x.to.txt: Failed to copy: close /data/Media2/Videos/Series/Series3/Season.2/Series3.S02E03.720p.HDTV.x264-AVS[PRiME]/Torrent Downloaded From 1337x.to.txt: input/output error
2017/03/06 07:30:07 NOTICE: Videos/Series/Series4/Season.2/Series4.S02E10.720p.webrip.h264-thumb.jpg: Removing partially written file on error: close /data/Media2/Videos/Series/Series4/Season.2/Series4.S02E10.720p.webrip.h264-thumb.jpg: input/output error
2017/03/06 07:30:07 ERROR : Videos/Series/Series4/Season.2/Series4.S02E10.720p.webrip.h264-thumb.jpg: Failed to remove partially written file: remove /data/Media2/Videos/Series/Series4/Season.2/Series4.S02E10.720p.webrip.h264-thumb.jpg: no such file or directory
2017/03/06 07:30:07 ERROR : Videos/Series/Series4/Season.2/Series4.S02E10.720p.webrip.h264-thumb.jpg: Failed to copy: close /data/Media2/Videos/Series/Series4/Season.2/Series4.S02E10.720p.webrip.h264-thumb.jpg: input/output error
2017/03/06 07:30:08 NOTICE: Videos/Series/Series4/tvshow.nfo: Removing partially written file on error: close /data/Media2/Videos/Series/Series4/tvshow.nfo: input/output error
2017/03/06 07:30:08 ERROR : Videos/Series/Series4/tvshow.nfo: Failed to remove partially written file: remove /data/Media2/Videos/Series/Series4/tvshow.nfo: no such file or directory
2017/03/06 07:30:08 ERROR : Videos/Series/Series4/tvshow.nfo: Failed to copy: close /data/Media2/Videos/Series/Series4/tvshow.nfo: input/output error
2017/03/06 07:30:08 NOTICE: Videos/Series/Series4/Season.2/Series4.S02E03.webrip.h264-thumb.jpg: Removing partially written file on error: close /data/Media2/Videos/Series/Series4/Season.2/Series4.S02E03.webrip.h264-thumb.jpg: input/output error
2017/03/06 07:30:08 ERROR : Videos/Series/Series4/Season.2/Series4.S02E03.webrip.h264-thumb.jpg: Failed to remove partially written file: remove /data/Media2/Videos/Series/Series4/Season.2/Series4.S02E03.webrip.h264-thumb.jpg: no such file or directory
2017/03/06 07:30:08 ERROR : Videos/Series/Series4/Season.2/Series4.S02E03.webrip.h264-thumb.jpg: Failed to copy: close /data/Media2/Videos/Series/Series4/Season.2/Series4.S02E03.webrip.h264-thumb.jpg: input/output error
2017/03/06 07:32:51 INFO  : Videos/Series/Series5/Season.6/Series5.S06E04.720p.hdtv.h264.mkv: Copied (new)
2017/03/06 07:34:58 INFO  :
Transferred:   3.713 GBytes (12.675 MBytes/s)

These things retry and sometimes they are successful and sometimes they are not. But overall there a LOT of errors/retries.

This is hte log from the mount itself

2017/03/06 07:30:07 ERROR : ....... : WriteFileHandle.Flush error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"9ea92930-0268-11e7-ad9f-43c113caf993\",\"message\":\"Rate exceeded\",\"code\":\"\"}"
2017/03/06 07:30:07 ERROR : ....... : Dir.Remove error: no such file or directory
2017/03/06 07:30:07 ERROR : ....... : WriteFileHandle.Release error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"9ecd2bf3-0268-11e7-99a8-477cb4e7d4b9\",\"message\":\"Rate exceeded\",\"code\":\"\"}"
2017/03/06 07:30:07 ERROR : ....... _HIDDEN~: WriteFileHandle.Release error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"9eedfaae-0268-11e7-a6cf-5d645af872ff\",\"message\":\"Rate exceeded\",\"code\":\"\"}"
2017/03/06 07:30:07 ERROR : ....... : WriteFileHandle.Flush error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"9f0a0daa-0268-11e7-9179-fbbb7ecb6809\",\"message\":\"Rate exceeded\",\"code\":\"\"}"
2017/03/06 07:30:07 ERROR : ....... : Dir.Remove error: no such file or directory
2017/03/06 07:30:08 ERROR :....... : WriteFileHandle.Flush error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"9f28b982-0268-11e7-ba9f-795965b51193\",\"message\":\"Rate exceeded\",\"code\":\"\"}"
2017/03/06 07:30:08 ERROR : ....... : Dir.Remove error: no such file or directory
2017/03/06 07:30:08 ERROR :....... : WriteFileHandle.Flush error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"9f1690d3-0268-11e7-8b59-6db6d047a573\",\"message\":\"Rate exceeded\",\"code\":\"\"}"
2017/03/06 07:30:08 ERROR : ....... : Dir.Remove error: no such file or directory
2017/03/06 07:38:24 ERROR : ....... : WriteFileHandle.Flush error: HTTP code 429: "429 Too Many Requests": response body: "{\"logref\":\"c639fc9a-0269-11e7-8580-85f63a4c128e\",\"message\":\"Rate exceeded\",\"code\":\"\"}"

I realize these errors are out of rclone’s control but something seems fishy with how poorly the mount performs errorwise compared to direct rclone copies to ACDgdrive. The actual BW usage is great. Its just the failed partial transfers retries thats the problem. I’d say 3/4 of my bandwidth is wasted with half uploaded files and then removals/tries sometimes ultimately failing and sometimes succeeding within the 3 retry allowances.

I can get a debug output if it would help. This just happened to be what I logged with regular logging for now.