rclone: gdrive server-side copy results in 404 errors

Reference thread:

https://forum.rclone.org/t/gdrive-to-gdrive-error-404-file-not-found/9621/10

When syncing two different gdrive remotes, i’m seeing 404 errors coming in the api and failures. When I disable the server-side copy the sync works correctly using “–disable copy”.

rclone sync robgs:cloudp/12el63s90885tg9u16iea5vtps/ao2c93qnjv533s8eoufko2p264/o59hc3g2a6qalng7k8mpmql28c pinagd:cloudp/12el63s90885tg9u16iea5vtps/ao2c93qnjv533s8eoufko2p264/o59hc3g2a6qalng7k8mpmql28c -vv --log-file=test.log --dump-bodies

I’ve run this with dump bodies just to see where the errors are coming from and here is the 404 and the info leading up to it.

2019/04/18 08:43:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/04/18 08:43:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/04/18 08:43:02 DEBUG : HTTP REQUEST (req 0xc000120d00)
2019/04/18 08:43:02 DEBUG : POST /drive/v3/files/1myq3teYU4hfZUtwLq_u9gYwaWm_M5FNO/copy?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink&keepRevisionForever=false&prettyPrint=false&supportsTeamDrives=false HTTP/1.1^M
Host: www.googleapis.com^M
User-Agent: rclone/v1.47.0-015-gaba15f11-beta^M
Content-Length: 215^M
Authorization: XXXX
Content-Type: application/json^M
Accept-Encoding: gzip^M
^M
{"description":"3i1g4iietqhggevatdgvq83v54","mimeType":"application/octet-stream","modifiedTime":"2019-03-27T20:03:42.648000000Z","name":"3i1g4iietqhggevatdgvq83v54","parents":["1dsH1bNv4_zrilBLIkFco49J0maPpxjl6"]}
2019/04/18 08:43:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/04/18 08:43:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/04/18 08:43:02 DEBUG : HTTP RESPONSE (req 0xc000120d00)
2019/04/18 08:43:02 DEBUG : HTTP/1.1 404 Not Found^M
Transfer-Encoding: chunked^M
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"^M
Cache-Control: private, max-age=0^M
Content-Type: application/json; charset=UTF-8^M
Date: Thu, 18 Apr 2019 12:43:02 GMT^M
Expires: Thu, 18 Apr 2019 12:43:02 GMT^M
Server: GSE^M
Vary: Origin^M
Vary: X-Origin^M
X-Content-Type-Options: nosniff^M
X-Frame-Options: SAMEORIGIN^M
X-Xss-Protection: 1; mode=block^M
^M
f6^M
{"error":{"errors":[{"domain":"global","reason":"notFound","message":"File not found: 1myq3teYU4hfZUtwLq_u9gYwaWm_M5FNO.","locationType":"parameter","location":"fileId"}],"code":404,"message":"File not found: 1myq3teYU4hfZUtwLq_u9gYwaWm_M5FNO."}}^M
0^M

root@cloudc:/data/bin# rclone lsjson robgs:cloudp/12el63s90885tg9u16iea5vtps/ao2c93qnjv533s8eoufko2p264/o59hc3g2a6qalng7k8mpmql28c
[
{"Path":"011nn2buif0v9dj091nd6u7sss","Name":"011nn2buif0v9dj091nd6u7sss","Size":5312,"MimeType":"application/octet-stream","ModTime":"2019-03-28T03:01:15.954Z","IsDir":false,"ID":"1qG1ucee_RkGWauRXzvsVRyjZ5s7ZouBg"},
{"Path":"3dfk7t6ff837dkm84kjviga1r8","Name":"3dfk7t6ff837dkm84kjviga1r8","Size":89623,"MimeType":"application/octet-stream","ModTime":"2019-03-27T17:11:10.804Z","IsDir":false,"ID":"1ZwK9Sem5PClKExyRmkA6_nM3ovn75SjV"},
{"Path":"3i1g4iietqhggevatdgvq83v54","Name":"3i1g4iietqhggevatdgvq83v54","Size":3309246,"MimeType":"application/octet-stream","ModTime":"2019-03-27T20:03:42.648Z","IsDir":false,"ID":"1myq3teYU4hfZUtwLq_u9gYwaWm_M5FNO"},
{"Path":"c562if70fm08du49pla2f9b2k8","Name":"c562if70fm08du49pla2f9b2k8","Size":95231,"MimeType":"application/octet-stream","ModTime":"2019-03-28T00:18:18.856Z","IsDir":false,"ID":"1RARk89nar7MwnY81sSV5oFnieobbGAwo"},
{"Path":"g8kg77r6lmb6tidovn7dg1a1g8","Name":"g8kg77r6lmb6tidovn7dg1a1g8","Size":6955,"MimeType":"application/octet-stream","ModTime":"2019-03-28T00:15:30.900Z","IsDir":false,"ID":"1sPfgXitLV2Ti9hRvFXoLrZALOFVSEA_K"},
{"Path":"gnr65g6hljq95f8f4uoldem888","Name":"gnr65g6hljq95f8f4uoldem888","Size":1072958,"MimeType":"application/octet-stream","ModTime":"2019-03-28T00:15:16.476Z","IsDir":false,"ID":"1lfYFSykIkv1J9afiufuVWSVLiAPAliQy"},
{"Path":"kbbmlqgruieun7ar0rm3a10tt0","Name":"kbbmlqgruieun7ar0rm3a10tt0","Size":2040,"MimeType":"application/octet-stream","ModTime":"2019-03-28T00:40:13.880Z","IsDir":false,"ID":"1mVr1b1wfNMpDbLPKYzwOEYXJgc8WEqym"},
{"Path":"obilalmb2cf58b6js0velfddqc","Name":"obilalmb2cf58b6js0velfddqc","Size":6121,"MimeType":"application/octet-stream","ModTime":"2019-03-28T02:16:16.786Z","IsDir":false,"ID":"1E71s4A3A2yRX7-ppnJEdDwIPbnty6HEa"},
{"Path":"ub0n2us8icde3qt4s9ihr96214","Name":"ub0n2us8icde3qt4s9ihr96214","Size":29644,"MimeType":"application/octet-stream","ModTime":"2019-03-28T03:01:18.722Z","IsDir":false,"ID":"1VZ74DLGpYEOn6Z2-NdOkV1toRg5V2EaN"},
{"Path":"uv710v07a8934pjtcjqtd6i3vg","Name":"uv710v07a8934pjtcjqtd6i3vg","Size":16834,"MimeType":"application/octet-stream","ModTime":"2019-03-28T00:40:16.532Z","IsDir":false,"ID":"1Y8goJZexNCqacqp_s0xoqzHwHpqPPsug"}


About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 49 (36 by maintainers)

Commits related to this issue

Most upvoted comments

My personal feeling is this really should be ‘opt-in’ as server side copies have far more stringent quotas than regular copies. But that is better answered by @ncw I think as whether it should be a opt-in or opt-out.

I’m tempted to revert back to the old behavior of only allowing server side copies in the same remote, unless the user provides a flag.

What does everyone else think?

In your reply to me did you mean server_side_across_accounts or server_side_across_configs? I think the latter.

Errm,yess

–drive-server-side-across-configs

Allow server side operations (eg copy) to work across different drive configs.

This can be useful if you wish to do a server side copy between two different Google drives. Note that this isn’t enabled by default because it isn’t easy to tell if it will work beween any two configurations.

  • Config: server_side_across_configs
  • Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS
  • Type: bool
  • Default: false

I could disable serverside for the whole session if it doesn’t have access to the other remote.

If i’m reading this correctly, won’t that mean that for each and every file that is part of a copy/move operation that it’ll fail first with a 404 and then switch to a regular copy? My concern is that the majority of users do not ‘share’ the folder between remotes which seems to be required to enable this to work so the majority of users by default will face many 404s and more API hits and slower operations because of the failure test.

Is my concern not correct?

For normal drive the destination need to be shared with, yes, either as everyone or direct share. But ‘add to my drive’ doesn’t create a copy, but instead links it.