maestral: Slow internet connection, parallel uploads time out, entire home network unusable

Describe the bug My internet connection is fairly slow, (about 2mbps up) and as a result, Maestral seems to get stuck in a loop and eat up bandwidth to the point of making the entire internet connection unusable when trying to upload a bunch of files at the same time. After monitoring with maestral activity, I noticed that it tries to upload over 30 files in parallel, all of them time out and do not complete, and then it restarts the whole process over again of uploading over 30 files in parallel, getting stuck in this never ending cycle.

Screen Shot 2021-08-17 at 11 38 23 PM

I’ve searched the docs to maybe find a way to configure maximum parallel uploads, but it doesn’t seem that there is such an option. My current workaround is to move all the files out of the folder, and drop them in one by one so that maestral doesn’t try to upload them all in parallel and crash my internet connection.

To Reproduce Drag a folder of over 100 images into maestral on a slow internet connection.

Expected behaviour Hopefully would be able to configure max parallel uploads to prevent this kind of problem.

System:

Maestral version: 1.4.6
Python version: 3.9.6
OS: MacOS Big Sur

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (10 by maintainers)

Commits related to this issue

[main, logging] rate limit how often status updates are propagated Unblocking on a future may be handled as a thread interrupt and some kernels, in particular Darwin, limit how often those are allowe... — committed to samschott/maestral by samschott a year ago

Most upvoted comments

Filed a feature request with the Dropbox SDK to allow chunked or streaming uploads: https://github.com/dropbox/dropbox-sdk-python/issues/459. If they agree drop the current limitation, this should greatly simplify implementing bandwidth usage limits.

samschott on Jan 22, 2023

If you are happy to look into this, a PR will be very welcome!

Bandwidth limits should apply across all parallel data transfers since a single upload or download can still max out the entire available bandwidth. We therefore ideally want to track total bandwidth usage over a sliding window (e.g., a few seconds) and rate limit chunked uploads or downloads accordingly.

The actual uploads and downloads are handled by the client module in client.DropboxClient.upload() and client.DropboxClient.download().

Downloads: We currently iterate over the response content in chunks of 8192 bytes. Rate limiting could easily plug in here and throttle / pause between iterations.

Uploads: Files that are larger than 5 MB are uploaded in chunks of 5 MB with each chunk being uploaded in a separate post request. That chunk size may already be too large for an effective bandwidth limit. However, I’m not sure if uploading smaller chunks with a larger number of requests is the best way to go here. Maybe we need a custom requests.Session object that implements rate limiting for the upload at a lower level. Any ideas you have a welcome! The relevant Dropbox API endpoint is files/upload_session/append which corresponds to files_upload_session_append_v2 in the Python SDK.

As you see, it won’t be an easy problem. I’m happy to help, especially with questions about the current code base.

samschott on Aug 21, 2021