rclone: FTP upload and download for big files (over 10GB) fails
Output of rclone version
rclone: Version “v1.55.1”
Describe the issue
This happens only for big files (over 10GB): the upload goes to 100% then rclone disconnects, reconnects to check the file on destination but declares that the upload has failed and restarts it! Adding --no-check-dest --low-level-retries 1
does not resolve the issue.
It works when uploading manually with the ftp
command! So the issue is NOT with the FTP server!
Typical rclone log/verbose going into a loop:
2021-05-13 18:34:22 DEBUG : big.zip: Need to transfer - File not found at Destination
2021-05-13 18:44:53 DEBUG : ftp://server.com:21/: Connecting to FTP server
2021-05-13 18:44:54 DEBUG : big.zip: Removed after failed upload: read tcp IP1:37706->IP2:21: i/o timeout
2021-05-13 18:44:54 DEBUG : big.zip: Received error: update stor: read tcp IP1:37706->IP2:21: i/o timeout - low level retry 1/10
I suspect the FTP library to be behind this bug: there is no timeout issue, I can see the file going to 100% on the FTP destination before it gets deleted by rclone. TLS is not used.
My suggestions/questions to fix: a) Why does rclone disconnects + reconnects to check the file instead of using the same FTP session? b) Can’t rclone sleeps for 2-3 seconds before doing the check of the uploaded file?
=> see log with --dump bodies
attached!
rclone_issue_5328.txt
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 27 (9 by maintainers)
I encountered similar problem and investigated. I’m now 100% sure we hit rclone bug with ftp timeouts!
@ncw bug description follows:
@ncw preliminary fix for:
@jcn50 Do you want to try experimental fix? It’s here: https://github.com/ivandeex/rclone/releases/tag/v1.56.0-ftp08