rclone: High memory usage when performing sync to azure blob storage
Output of rclone version
$ ./rclone version rclone v1.43-136-g87e1efa9-beta
- os/arch: linux/amd64
- go version: go1.11.1
Describe the issue
Based on this forum thread: https://forum.rclone.org/t/high-memory-usage-when-performing-sync/7085/2
I’m encountering some pretty high memory usage when performing a sync from the local filesystem to a remote encrypted fs (which itself wraps an azure blob storage account). I suspect this may be because I have quite a lot of smallish files to sync. Key info:
Command being run: rclone --config /path/to/my/version/controlled/rclone.conf --checksum --retries 20 --suffix 2018-10-04 --backup-dir azure-encrypted:/backups.history/ --quiet --bwlimit 384k sync /home/backup-user/backups/ azure-encrypted:/backups
Rclone Version: 1.42 (also repros on latest beta, v1.43-136-g87e1efa9-beta) Total size of files being sync’ed: 855GB Number of files being sync’ed: 1,098,342 (1098342 without locale-specific separators)
The backup is using 5.325GB of physical RAM (12.408GB virtual), which seems pretty extreme.
Profiling data:
(pprof) text
Showing nodes accounting for 2268.21MB, 84.14% of 2695.62MB total
Dropped 117 nodes (cum <= 13.48MB)
Showing top 10 nodes out of 104
flat flat% sum% cum cum%
960.02MB 35.61% 35.61% 960.02MB 35.61% encoding/xml.copyValue
608.13MB 22.56% 58.17% 611.13MB 22.67% github /ncw/rclone/vendor/github.com/Azure/azure-storage-blob-go/2018-03-28/azblob.(*Metadata).UnmarshalXML
168.52MB 6.25% 64.43% 224.02MB 8.31% github /ncw/rclone/backend/azureblob.(*Fs).newObjectWithInfo
117.02MB 4.34% 68.77% 117.02MB 4.34% strings.Join
81MB 3.00% 71.77% 81MB 3.00% github /ncw/rclone/vendor/github.com/rfjakob/eme.tabulateL
79.50MB 2.95% 74.72% 79.50MB 2.95% strings.ToUpper
74.02MB 2.75% 77.47% 74.02MB 2.75% github /ncw/rclone/fs/asyncreader.newBuffer
72MB 2.67% 80.14% 153MB 5.68% github /ncw/rclone/vendor/github.com/rfjakob/eme.Transform
55.50MB 2.06% 82.20% 55.50MB 2.06% encoding/base64.(*Encoding).EncodeToString
52.50MB 1.95% 84.14% 52.50MB 1.95% encoding/base32.(*Encoding).DecodeString
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 28 (14 by maintainers)
My apologies for delay on this, its been a bit hectic week, I’ve not yet got to the bottom of this issue, will try to have an update by tomorrow, based on minor tests, I feel like this is issue at client library it self, if that’s the case I’ll follow up with azure team.
This is with rclone-v1.44-071-g9322f4ba-beta-linux-amd64
@OliverEvans96 wrote:
This issue is definitely a problem with Azure blob storage.
If you think there is a new problem then can you try again with the latest beta and if it shows the problem could you follow the instructions here: https://rclone.org/rc/#debugging-rclone-with-pprof - run the sync for a while with the
--rcparameter then do a memory profile.Post the text or graphical output of that in please a new issue on github and I’ll take a look - thanks