rclone: Google Photos backend hanging after corpus has grown sufficiently large

What is the problem you are having with rclone?

I’m using rclone to sync photos from local storage into Google Photos and I’m only interested in a one-way sync (but including deletes). This was working fine in filling a backlog of photos over the last year or so.

I now have 300+GB of photos on Google Photos, but after adding a couple thousand more photos to upload, rclone now simply “hangs” when syncing. It is making no more progress, but still posts it’s progress update every once in a while and after 72h or so it sometimes exits.

The same can be reproduced simply by using rclone ls -v -v -v -v -v remote:album. It will run and make sufficient progress, and then it enters this loop:

2022/08/22 13:19:47 DEBUG : 2015-03--05/some file.jpg: Size: 
2022/08/22 13:19:47 DEBUG : 2015-03--05/some file.jpg: >Size: 
       -1 2015-03--05/some file.jpg
2022/08/22 13:27:22 DEBUG : remote: Loaded invalid token from config file - ignoring
2022/08/22 13:27:23 DEBUG : Saving config "token" in section "remote" of the config file
2022/08/22 13:27:23 DEBUG : remote: Saved new token in config file
2022/08/22 14:27:12 DEBUG : remote: Loaded invalid token from config file - ignoring
2022/08/22 14:27:12 DEBUG : Saving config "token" in section "remote" of the config file
2022/08/22 14:27:12 DEBUG : remote: Saved new token in config file
2022/08/22 15:27:01 DEBUG : remote: Loaded invalid token from config file - ignoring
2022/08/22 15:27:02 DEBUG : Saving config "token" in section "remote" of the config file
2022/08/22 15:27:02 DEBUG : remote: Saved new token in config file
2022/08/22 16:01:43 DEBUG : pacer: low level retry 1/10 (error read tcp [XX:XX:XX::1]:31653->[2a00:1450:400a:802::200a]:443: read: connection reset by peer)
2022/08/22 16:01:43 DEBUG : pacer: Rate limited, increasing sleep to 1.428416024s
2022/08/22 16:01:43 DEBUG : pacer: Reducing sleep to 0s
2022/08/22 16:26:51 DEBUG : remote: Loaded invalid token from config file - ignoring
2022/08/22 16:26:51 DEBUG : Saving config "token" in section "remote" of the config file
2022/08/22 16:26:51 DEBUG : remote: Saved new token in config file
2022/08/22 17:26:40 DEBUG : remote: Loaded invalid token from config file - ignoring
2022/08/22 17:26:40 DEBUG : Saving config "token" in section "remote" of the config file
2022/08/22 17:26:40 DEBUG : remote: Saved new token in config file
2022/08/22 18:26:29 DEBUG : remote: Loaded invalid token from config file - ignoring
2022/08/22 18:26:29 DEBUG : Saving config "token" in section "remote" of the config file
2022/08/22 18:26:29 DEBUG : remote: Saved new token in config file

Is there a way to further increase the log level?

Running lsd is not affected, I guess because the volume of data is small.

What is your rclone version (output from rclone version)

% rclone --version
rclone v1.58.1-DEV
- os/version: freebsd 13.1-stable (64 bit)
- os/kernel: 13.1-stable (amd64)
- os/type: freebsd
- os/arch: amd64
- go/version: go1.18.3
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

FreeBSD, 64bit

Which cloud storage system are you using? (e.g. Google Drive)

Google Photos

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone ls -v -v -v -v -v remote:album

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

see relevant output above.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don’t comment if you have no relevant information to add. It’s just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.