rclone: Memory leak when performing sync to Google clould storage with client-side encryption

What is the problem you are having with rclone?

On Linux, when rclone performs sync to google cloud storage with client-side encryption, “top” shows that the memory usage for rclone keeps going up. Eventaully rclone encounters “fatal error: out of memory” and quits. rclone memory profiling confirms that problem, and points to “encoding/json.(*decodeState).literalStore” being the primary suspect.

What is your rclone version (output from rclone version)

rclone v1.47.0-067-g9c6f3ae8-beta

• os/arch: linux/amd64
• go version: go1.12.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 6.6, with Linux 2.6.32-504.3.3.el6.x86_64 with 16GB RAM

Which cloud storage system are you using? (eg Google Drive)

Google cloud storage, with client-side encryption

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --transfers=2 --cache-chunk-no-memory sync /xxxx gcs-sa-enc:/xxxx --rc

Configuration file rclone.conf:

[gcs-sa]
type = google cloud storage
project_number = xxxx
service_account_file = xxxx
storage_class = COLDLINE
object_acl = private
bucket_acl = private
client_id = xxxx

[gcs-sa-enc]
type = crypt
remote = gcs-sa:xxxx/xxxx
filename_encryption = standard
directory_name_encryption = true
password = xxxx
password2 = xxxx

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

$ go tool pprof -text http://localhost:5572/debug/pprof/heap
Run above profiling command periodically, with following outputs

Time: May 13, 2019 at 11:21pm (PDT)
Showing nodes accounting for 549.92MB, 98.15% of 560.31MB total
Dropped 64 nodes (cum <= 2.80MB)
      flat  flat%   sum%        cum   cum%
  446.69MB 79.72% 79.72%   446.69MB 79.72%  encoding/json.(*decodeState).literalStore
   51.51MB  9.19% 88.92%    65.51MB 11.69%  github.com/ncw/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
      24MB  4.28% 93.20%       24MB  4.28%  github.com/ncw/rclone/vendor/google.golang.org/api/gensupport.NewMediaBuffer
      14MB  2.50% 95.70%       14MB  2.50%  encoding/hex.EncodeToString
    6.94MB  1.24% 96.94%    72.44MB 12.93%  github.com/ncw/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
       4MB  0.71% 97.65%        4MB  0.71%  strings.(*Builder).grow
    2.28MB  0.41% 98.06%     3.28MB  0.59%  github.com/ncw/rclone/fs/sync.(*syncCopyMove).DstOnly
    0.50MB 0.089% 98.15%     4.48MB   0.8%  github.com/ncw/rclone/fs/march.(*March).processJob
         0     0% 98.15%   448.73MB 80.09%  encoding/json.(*Decoder).Decode

Time: May 13, 2019 at 11:28pm (PDT)
Showing nodes accounting for 996.43MB, 98.54% of 1011.23MB total
Dropped 78 nodes (cum <= 5.06MB)
      flat  flat%   sum%        cum   cum%
  817.85MB 80.88% 80.88%   817.85MB 80.88%  encoding/json.(*decodeState).literalStore
   90.51MB  8.95% 89.83%   118.51MB 11.72%  github.com/ncw/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
      32MB  3.16% 92.99%       32MB  3.16%  github.com/ncw/rclone/vendor/google.golang.org/api/gensupport.NewMediaBuffer
      28MB  2.77% 95.76%       28MB  2.77%  encoding/hex.EncodeToString
   13.55MB  1.34% 97.10%   132.07MB 13.06%  github.com/ncw/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
    9.50MB  0.94% 98.04%     9.50MB  0.94%  strings.(*Builder).grow
    4.52MB  0.45% 98.49%    10.02MB  0.99%  github.com/ncw/rclone/fs/sync.(*syncCopyMove).DstOnly
    0.50MB  0.05% 98.54%    10.52MB  1.04%  github.com/ncw/rclone/fs/march.(*March).processJob
         0     0% 98.54%   821.58MB 81.25%  encoding/json.(*Decoder).Decode

	Time: May 14, 2019 at 12:07am (PDT)
Showing nodes accounting for 3253.22MB, 98.90% of 3289.37MB total
Dropped 109 nodes (cum <= 16.45MB)
      flat  flat%   sum%        cum   cum%
 2610.13MB 79.35% 79.35%  2610.13MB 79.35%  encoding/json.(*decodeState).literalStore
  299.53MB  9.11% 88.46%   375.53MB 11.42%  github.com/ncw/rclone/backend/googlecloudstorage.(*Fs).newObjectWithInfo
   91.50MB  2.78% 91.24%    91.50MB  2.78%  github.com/ncw/rclone/backend/crypt.(*Fs).newObject
      76MB  2.31% 93.55%       76MB  2.31%  encoding/hex.EncodeToString
   51.74MB  1.57% 95.12%   427.28MB 12.99%  github.com/ncw/rclone/backend/googlecloudstorage.(*Fs).listDir.func1
      30MB  0.91% 96.03%       30MB  0.91%  github.com/ncw/rclone/vendor/github.com/rfjakob/eme.tabulateL
   27.76MB  0.84% 96.88%    27.76MB  0.84%  github.com/ncw/rclone/lib/pool.New.func1
      27MB  0.82% 97.70%       27MB  0.82%  strings.(*Builder).grow
      18MB  0.55% 98.25%       48MB  1.46%  github.com/ncw/rclone/vendor/github.com/rfjakob/eme.Transform
   13.55MB  0.41% 98.66%    20.55MB  0.62%  github.com/ncw/rclone/fs/sync.(*syncCopyMove).DstOnly
    4.50MB  0.14% 98.79%       73MB  2.22%  github.com/ncw/rclone/backend/crypt.(*cipher).decryptSegment
    2.50MB 0.076% 98.87%    20.50MB  0.62%  github.com/ncw/rclone/backend/crypt.decodeFileName
    1.01MB 0.031% 98.90%    21.56MB  0.66%  github.com/ncw/rclone/fs/march.(*March).processJob
         0     0% 98.90%  2610.66MB 79.37%  encoding/json.(*Decoder).Decode

Eventually rclone ran out of memory and crashed.