rclone: FTP upload and download for big files (over 10GB) fails

Output of rclone version

rclone: Version “v1.55.1”

Describe the issue

This happens only for big files (over 10GB): the upload goes to 100% then rclone disconnects, reconnects to check the file on destination but declares that the upload has failed and restarts it! Adding --no-check-dest --low-level-retries 1 does not resolve the issue.

It works when uploading manually with the ftp command! So the issue is NOT with the FTP server!

Typical rclone log/verbose going into a loop:

2021-05-13 18:34:22 DEBUG : big.zip: Need to transfer - File not found at Destination
2021-05-13 18:44:53 DEBUG : ftp://server.com:21/: Connecting to FTP server
2021-05-13 18:44:54 DEBUG : big.zip: Removed after failed upload: read tcp IP1:37706->IP2:21: i/o timeout
2021-05-13 18:44:54 DEBUG : big.zip: Received error: update stor: read tcp IP1:37706->IP2:21: i/o timeout - low level retry 1/10

I suspect the FTP library to be behind this bug: there is no timeout issue, I can see the file going to 100% on the FTP destination before it gets deleted by rclone. TLS is not used.

My suggestions/questions to fix: a) Why does rclone disconnects + reconnects to check the file instead of using the same FTP session? b) Can’t rclone sleeps for 2-3 seconds before doing the check of the uploaded file?

=> see log with --dump bodies attached! rclone_issue_5328.txt

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 27 (9 by maintainers)

Most upvoted comments

I encountered similar problem and investigated. I’m now 100% sure we hit rclone bug with ftp timeouts!

@ncw bug description follows:

  • jlaffaye/ftp makes control/data streams from a factory provided by rclone. this factory wraps go tcp streams in the code which cancels streams after timeout but resets timeout timer after each read/write.
  • a typical ftp upload is: client talks with ftp server on control stream, sets upload parameters, then create one or more data streams and do the upload (this can be long), then return to control stream and ask ftp server about upload results (here be dragons)
  • while client was busy uploading on data stream, the control stream timeout happened and client’s attempt to read status returned I/O timeout

@ncw preliminary fix for:

@jcn50 Do you want to try experimental fix? It’s here: https://github.com/ivandeex/rclone/releases/tag/v1.56.0-ftp08