lakeFS: RClone and LakeFS integration breaks without v2_auth

This might be intended and just needs an update on your website but I ran into a hard to debug issue trying to sync data into LakeFS with RClone. Note, I’ve followed the instructions previously on the site and this worked so not sure what LakeFS version it broke in. The behavior seems to point towards the client side however I haven’t updated RClone at all since it did previously work.

LakeFS Version: 0.48.0 (Also reproduced on 0.47.0) RClone Version:

rclone v1.56.0
- os/version: darwin 11.5.2 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.6
- go/linking: dynamic
- go/tags: none

Error trying to copy local data into LakeFS using the suggested configuration:

.... removed ...

2021/09/02 21:07:25 INFO  :
Transferred:   	   14.481Ki / 14.481 KiByte, 100%, 0 Byte/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         1.2s

2021/09/02 21:07:25 Failed to sync with 2 errors: last error was: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource></Resource><Region>us-gov-west-1</Region><RequestId>5d580752-d488-4f4f-976b-358729075279</RequestId><HostId>D071307137586F18</HostId></Error>

To test that the credentials were correct I went the other way and used LakeFS as a source after adding a file via the UI

❯ rclone ls lakefs:aif-xxxx/main/ -vv

2021/09/02 21:10:46 DEBUG : Setting --ca-cert "/Users/e379822/certs/lm_ca.pem" from environment variable RCLONE_CA_CERT="/Users/e379822/certs/lm_ca.pem"
2021/09/02 21:10:46 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "ls" "lakefs:aif-xxxx/main/" "-vv"]
2021/09/02 21:10:46 DEBUG : Creating backend with remote "lakefs:aif-xxxx/main/"
2021/09/02 21:10:46 DEBUG : Using config file from "/Users/e379822/.config/rclone/rclone.conf"
2021/09/02 21:10:46 DEBUG : fs cache: renaming cache item "lakefs:aif-xxxx/main/" to be canonical "lakefs:aif-xxxx/main"
        4 test.txt
2021/09/02 21:10:47 DEBUG : 6 go routines active

What put me onto it was the signature signing is this log line in the LakeFS stating it was using SigV4. time="2021-09-03T01:51:31Z" level=warning msg="error verifying credentials for key" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:54" authenticator=sigv4 error=SignatureDoesNotMatch key=AKIAJ6UDLXIPOISF7LKQ

Also verified by dumping the RClone headers that it was using SigV4 authentication by dumping the headers.

2021/09/02 20:56:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 20:56:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 20:56:06 DEBUG : HTTP REQUEST (req 0xc000ad0100)
2021/09/02 20:56:06 DEBUG : PUT /aif-xxxx/main/test/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ6UDLXIPOISF7LKQ%2F20210903%2Fus-gov-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210903T025606Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=c3b12b3bb96069f6102df42eb22b4a64d7bc728800e53fee1b6372547710fdeb HTTP/1.1
Host: s3.lakefs.ai.us.lmco.com:443
User-Agent: rclone/v1.56.0
Content-Length: 4
content-md5: uh8lEfwwQjvbsYP+M/PdDw==
content-type: text/plain; charset=utf-8
x-amz-acl: private
x-amz-meta-mtime: 1630636753.967220656
Accept-Encoding: gzip

Setting the V2 auth in rclone does fix this issue:

❯ rclone sync -v test lakefs:aif-xxxx/main/test/ --s3-v2-auth --dump headers

....

2021/09/02 21:16:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 21:16:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 21:16:33 DEBUG : HTTP REQUEST (req 0xc0005a4200)
2021/09/02 21:16:33 DEBUG : PUT /aif-xxxxx/main/test/test.txt HTTP/1.1
Host: s3.lakefs.ai.us.lmco.com:443
User-Agent: rclone/v1.56.0
Content-Length: 4
Authorization: XXXX
Content-Md5: uh8lEfwwQjvbsYP+M/PdDw==
Content-Type: text/plain; charset=utf-8
Date: Fri, 03 Sep 2021 03:16:33 UTC
X-Amz-Acl: private
X-Amz-Meta-Mtime: 1630636753.967220656
Accept-Encoding: gzip

.... 
 
2021/09/02 21:16:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/02 21:16:34 DEBUG : test.txt: md5 = ba1f2511fc30423bdbb183fe33f3dd0f OK
2021/09/02 21:16:34 INFO  : test.txt: Copied (new)
2021/09/02 21:16:34 DEBUG : Waiting for deletions to finish
2021/09/02 21:16:34 INFO  :
Transferred:   	    4.827Ki / 4.827 KiByte, 100%, 0 Byte/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         1.5s

2021/09/02 21:16:34 DEBUG : 9 go routines active

It’s an easy enough work around to add v2_auth = true, however I wanted to report it in case it is a bug since you have SigV4 listed as supported here