gphotos-sync: Sudden FAILURE for every download

First of all, thanks for this great project. 😃

After downloading 75,450 photos and videos successfully, every attempt is now returning an instant FAILURE. The quotas on the Google Console only show about 30% usage, but to rule this out, I created a new client ID with no quota usage at all, and it keeps happening.

Unfortunately, even with a trace level, nothing gets appended to gphotos.trace for these downloading attempts. If I search for these files on photos.google.com I can see and download them perfectly fine. I’ve also made sure the partition still has free space (3.7 TB) and I can write files perfectly fine.

Running version: 2.14.0, database schema version 5.7.

Any tips on how to debug this?

02-26 11:58:34 gphotos.GooglePhotosDownload INFO     downloading 1 photos/2017/09/IMG_0884.MOV
02-26 11:58:34 gphotos.GooglePhotosDownload ERROR    FAILURE 1 downloading photos/2017/09/IMG_0884.MOV
02-26 11:58:34 gphotos.GooglePhotosDownload INFO     downloading 2 photos/2016/09/IMG_0200 (2).JPG
02-26 11:58:35 gphotos.GooglePhotosDownload ERROR    FAILURE 2 downloading photos/2016/09/IMG_0200 (2).JPG
02-26 11:58:35 gphotos.GooglePhotosDownload INFO     downloading 3 photos/2016/09/IMG_0172 (2).JPG
02-26 11:58:36 gphotos.GooglePhotosDownload ERROR    FAILURE 3 downloading photos/2016/09/IMG_0172 (2).JPG
02-26 11:58:36 gphotos.GooglePhotosDownload INFO     downloading 4 photos/2016/09/IMG_0154 (2).JPG
02-26 11:58:37 gphotos.GooglePhotosDownload ERROR    FAILURE 4 downloading photos/2016/09/IMG_0154 (2).JPG
02-26 11:58:37 gphotos.GooglePhotosDownload INFO     downloading 5 photos/2016/09/IMG_0151 (2).JPG
02-26 11:58:38 gphotos.GooglePhotosDownload ERROR    FAILURE 5 downloading photos/2016/09/IMG_0151 (2).JPG
02-26 11:58:38 gphotos.GooglePhotosDownload INFO     downloading 6 photos/2016/09/IMG_0139 (2).JPG
02-26 11:58:39 gphotos.GooglePhotosDownload ERROR    FAILURE 6 downloading photos/2016/09/IMG_0139 (2).JPG
02-26 11:58:39 gphotos.GooglePhotosDownload INFO     downloading 7 photos/2016/09/IMG_0128.JPG
02-26 11:58:40 gphotos.GooglePhotosDownload ERROR    FAILURE 7 downloading photos/2016/09/IMG_0128.JPG
02-26 11:58:40 gphotos.GooglePhotosDownload INFO     downloading 8 photos/2016/09/IMG_0125 (2).JPG
02-26 11:58:41 gphotos.GooglePhotosDownload ERROR    FAILURE 8 downloading photos/2016/09/IMG_0125 (2).JPG
02-26 11:58:41 gphotos.GooglePhotosDownload INFO     downloading 9 photos/2016/09/IMG_0118 (2).JPG
02-26 11:58:42 gphotos.GooglePhotosDownload ERROR    FAILURE 9 downloading photos/2016/09/IMG_0118 (2).JPG
02-26 11:58:42 gphotos.GooglePhotosDownload INFO     downloading 10 photos/2016/09/IMG_0099 (2).JPG
02-26 11:58:43 gphotos.GooglePhotosDownload ERROR    FAILURE 10 downloading photos/2016/09/IMG_0099 (2).JPG
02-26 11:58:43 gphotos.GooglePhotosDownload INFO     downloading 11 photos/2016/09/IMG_0092 (2).JPG
02-26 11:58:43 gphotos.GooglePhotosDownload ERROR    FAILURE 11 downloading photos/2016/09/IMG_0092 (2).JPG
02-26 11:58:43 gphotos.GooglePhotosDownload INFO     downloading 12 photos/2016/09/IMG_0088 (2).JPG
02-26 11:58:44 gphotos.GooglePhotosDownload ERROR    FAILURE 12 downloading photos/2016/09/IMG_0088 (2).JPG
02-26 11:58:44 gphotos.GooglePhotosDownload INFO     downloading 13 photos/2016/09/IMG_0073 (2).JPG
02-26 11:58:45 gphotos.GooglePhotosDownload ERROR    FAILURE 13 downloading photos/2016/09/IMG_0073 (2).JPG
02-26 11:58:45 gphotos.GooglePhotosDownload INFO     downloading 14 photos/2016/09/IMG_0066.JPG
02-26 11:58:46 gphotos.GooglePhotosDownload ERROR    FAILURE 14 downloading photos/2016/09/IMG_0066.JPG
02-26 11:58:46 gphotos.GooglePhotosDownload INFO     downloading 15 photos/2016/09/IMG_0063 (2).JPG
02-26 11:58:47 gphotos.GooglePhotosDownload ERROR    FAILURE 15 downloading photos/2016/09/IMG_0063 (2).JPG
02-26 11:58:47 gphotos.GooglePhotosDownload INFO     downloading 16 photos/2016/09/IMG_0062 (2).JPG
02-26 11:58:48 gphotos.GooglePhotosDownload ERROR    FAILURE 16 downloading photos/2016/09/IMG_0062 (2).JPG
02-26 11:58:48 gphotos.GooglePhotosDownload INFO     downloading 17 photos/2016/09/IMG_0054 (2).JPG
02-26 11:58:49 gphotos.GooglePhotosDownload ERROR    FAILURE 17 downloading photos/2016/09/IMG_0054 (2).JPG
02-26 11:58:49 gphotos.GooglePhotosDownload INFO     downloading 18 photos/2016/09/IMG_0044 (2).JPG
02-26 11:58:50 gphotos.GooglePhotosDownload ERROR    FAILURE 18 downloading photos/2016/09/IMG_0044 (2).JPG
02-26 11:58:50 gphotos.GooglePhotosDownload INFO     downloading 19 photos/2016/09/IMG_0043 (2).JPG
02-26 11:58:51 gphotos.GooglePhotosDownload ERROR    FAILURE 19 downloading photos/2016/09/IMG_0043 (2).JPG
02-26 11:58:51 gphotos.GooglePhotosDownload INFO     downloading 20 photos/2016/09/IMG_0042 (2).JPG
02-26 11:58:52 gphotos.GooglePhotosDownload ERROR    FAILURE 20 downloading photos/2016/09/IMG_0042 (2).JPG
02-26 11:58:52 gphotos.GooglePhotosDownload INFO     downloading 21 photos/2016/09/IMG_0019 (2).JPG
02-26 11:58:53 gphotos.GooglePhotosDownload ERROR    FAILURE 21 downloading photos/2016/09/IMG_0019 (2).JPG
02-26 11:58:53 gphotos.GooglePhotosDownload INFO     downloading 22 photos/2016/09/IMG_0012.JPG
02-26 11:58:54 gphotos.GooglePhotosDownload ERROR    FAILURE 22 downloading photos/2016/09/IMG_0012.JPG
02-26 11:58:54 gphotos.GooglePhotosDownload INFO     downloading 23 photos/2016/09/IMG_3068.MOV
02-26 11:58:54 gphotos.GooglePhotosDownload ERROR    FAILURE 23 downloading photos/2016/09/IMG_3068.MOV
02-26 11:58:54 gphotos.GooglePhotosDownload INFO     downloading 24 photos/2016/09/IMG_3062.PNG
02-26 11:58:54 gphotos.GooglePhotosDownload ERROR    FAILURE 24 downloading photos/2016/09/IMG_3062.PNG
02-26 11:58:54 gphotos.GooglePhotosDownload INFO     downloading 25 photos/2016/08/IMG_3028.JPG
02-26 11:58:55 gphotos.GooglePhotosDownload ERROR    FAILURE 25 downloading photos/2016/08/IMG_3028.JPG
02-26 11:58:55 gphotos.GooglePhotosDownload INFO     downloading 26 photos/2016/08/IMG_3014.JPG
02-26 11:58:55 gphotos.GooglePhotosDownload ERROR    FAILURE 26 downloading photos/2016/08/IMG_3014.JPG

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 26 (15 by maintainers)

Most upvoted comments

The latest push of the code in the branch 429-reponse handles backoff correctly. It reads the ‘retry after’ response header to get the server’s recommendation on how long to pause.

Not much credit due to me since it turns out I just needed to switch on the feature which is in the python urllib3 library.

I’m not sure if you are interested in gphotos-sync anymore now you are aware of the video transcoding issue. But if you are then please can you try a little stress test?

Thanks, giles.

gilesknap on Feb 29, 2020

OK, that makes sense. I’ll add the backoff soon.

gilesknap on Feb 28, 2020