minio-go: Unable to access bucket with restricted account in eu-central-1

Hi,

I have a somewhat peculiar problem. A user contributed a tutorial to setup a restricted account with S3 which can just access one bucket (and e.g. not create new buckets or access the console). Since I’ve updated minio-go in restic for https://github.com/minio/minio/issues/4275 I’m unable to create new repos via this restricted account (it worked before).

The error is (including the trace, built with minio-go 5d7ee332f62e83d36beba669be671801180abb89):

$ restic -r s3:s3.amazonaws.com/restic-test-travis/x2 init

---------START-HTTP---------                                   
GET /restic-test-travis/?location= HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/2.1.0
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJSQEUXZSP56YBL6Q/20170512/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=19ac77e663ecaee24ab7832479152865fc71ac5a06bf33914dcca617b59c6fc7
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20170512T203950Z
Accept-Encoding: gzip

HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 12 May 2017 20:39:51 GMT
Server: AmazonS3
X-Amz-Id-2: 5jNddocga32YohOP6BpvdE2/OxKFm1gOSiznLvdhXz6/ux5R3dAq1qoNQLTGswFzNZkafUtSMNI=
X-Amz-Request-Id: 31AD0B0559C5E892

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>31AD0B0559C5E892</RequestId><HostId>5jNddocga32YohOP6BpvdE2/OxKFm1gOSiznLvdhXz6/ux5R3dAq1qoNQLTGswFzNZkafUtSMNI=</HostId></Error>
0
---------END-HTTP---------
---------START-HTTP---------
HEAD / HTTP/1.1
Host: restic-test-travis.s3.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/2.1.0
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJSQEUXZSP56YBL6Q/20170512/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d87e750f8de5e6d21e8ceffca75b095eed69ff43744a1ac3f3e3a12029b70855
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20170512T203951Z

HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 12 May 2017 20:39:51 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: 0qQw45YtBWvgSUm5HFoQ8UJ2w/XWn/KRpZyg6xj5DDMC7kPHhOFxVnIfM6OozkqvHOrVojoU49k=
X-Amz-Request-Id: AE1F2FC4AA4843B1

---------END-HTTP---------
create backend at s3:s3.amazonaws.com/restic-test-travis/x2 failed: client.BucketExists: 400 Bad Request

I’ve done a git bisect from b1674741d196d5d79486d7c1645ed6ded902b712 (good) to 5d7ee332f62e83d36beba669be671801180abb89 (bad), and the commit that caused it was:

fd942284fe190615d098af0b63e1698ef0969df1 is the first bad commit                                          
commit fd942284fe190615d098af0b63e1698ef0969df1                                                           
Author: Harshavardhana <harsha@minio.io>                                                                  
Date:   Tue Apr 4 11:35:20 2017 -0700                                                                     
                                                                                                          
    api: Check for Code 'InvalidRegion' for retrying with server Region. (#639)                           
                                                                                                          
diff --git a/api.go b/api.go
index 2beba77..e971721 100644
--- a/api.go
+++ b/api.go
@@ -543,9 +543,9 @@ func (c Client) executeMethod(method string, metadata requestMetadata) (res *htt
 
        // For errors verify if its retryable otherwise fail quickly.
        errResponse := ToErrorResponse(httpRespToErrorResponse(res, metadata.bucketName, metadata.objectNa
-       // Bucket region if set in error response, we can retry the
-       // request with the new region.
-       if errResponse.Region != "" {
+       // Bucket region if set in error response and the error code dictates invalid region,
+       // we can retry the request with the new region.
+       if errResponse.Code == "InvalidRegion" && errResponse.Region != "" {
            c.bucketLocCache.Set(metadata.bucketName, errResponse.Region)
            continue // Retry.
        }

On the other hand, a successful trace (with the commit right before that one) is:

---------START-HTTP---------                               
GET /restic-test-travis/?location= HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/2.0.4
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20170512/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20170512T203041Z
Accept-Encoding: gzip

HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 12 May 2017 20:30:41 GMT
Server: AmazonS3
X-Amz-Id-2: wFRgmf4iQpCw/spDBjf3kh6pZPCQ91bnrvBZGMs29ZqD3fLJfaHnoh0qtB1LphAv09Z67UKy0F8=
X-Amz-Request-Id: 9D1AEFF328416280

f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>9D1AEFF328416280</RequestId><HostId>wFRgmf4iQpCw/spDBjf3kh6pZPCQ91bnrvBZGMs29ZqD3fLJfaHnoh0qtB1LphAv09Z67UKy0F8=</HostId></Error>
0
---------END-HTTP---------
---------START-HTTP---------
HEAD / HTTP/1.1
Host: restic-test-travis.s3.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/2.0.4
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20170512/us-east-1/s3/aws4_request, SignedHeaders=expect;host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
Expect: 100-continue
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20170512T203041Z

HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 12 May 2017 20:30:41 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: yixxZF7DxK/3bJL9qlkrgHzjG7qDha2VrAtnhNLjezsoywZcjoxs9IsGhMN/wepfxOg++ekxou4=
X-Amz-Request-Id: 1C93A9104ED73A7D

---------END-HTTP---------
---------START-HTTP---------
HEAD / HTTP/1.1
Host: restic-test-travis.s3-eu-central-1.amazonaws.com
User-Agent: Minio (linux; amd64) minio-go/2.0.4
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20170512/eu-central-1/s3/aws4_request, SignedHeaders=expect;host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
Expect: 100-continue
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20170512T203041Z

HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 12 May 2017 20:30:43 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-central-1
X-Amz-Id-2: NT+/zXZ+hTi+iAKapHhlHByQ4a1I+ifFo3vbAdQBZ4Q3NEoyxgDxBIxqbHgmC/oWHPESv8S8mDc=
X-Amz-Request-Id: AB5769CE0F7D8CA6

---------END-HTTP---------

It seems to me that testing for errResponse.Code == "InvalidRegion" is not enough: It should also test for a response code of 400 Bad Request when a region is present, and try that. I’ve verified that the following patch works:

diff --git a/api.go b/api.go
index 9cd2b6d..6beb445 100644
--- a/api.go
+++ b/api.go
@@ -556,7 +556,7 @@ func (c Client) executeMethod(method string, metadata requestMetadata) (res *htt
 		// Bucket region if set in error response and the error
 		// code dictates invalid region, we can retry the request
 		// with the new region.
-		if errResponse.Code == "InvalidRegion" && errResponse.Region != "" {
+		if errResponse.Region != "" && (errResponse.Code == "InvalidRegion" || errResponse.Code == "400 Bad Request") {
 			c.bucketLocCache.Set(metadata.bucketName, errResponse.Region)
 			continue // Retry.
 		}

Maybe this also needs to be added to api-put-bucket.go?

FYI: I’m not sure if errResponse.Code will ever by equal to InvalidRegion, since in all cases I observed the string always started with the numeric HTTP response code and it also contained spaces…

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 24 (24 by maintainers)

Commits related to this issue

api: Failed call retry with region only when http.StatusBadRequest. Fixes #677 — committed to harshavardhana/minio-go by minio-trusted 7 years ago
api: Failed call retry with region only when http.StatusBadRequest. Fixes #677 — committed to harshavardhana/minio-go by harshavardhana 7 years ago
minio: Apply fix https://github.com/minio/minio-go/issues/677 — committed to restic/restic by fd0 7 years ago
api: Failed call retry with region only when http.StatusBadRequest. Fixes #677 — committed to harshavardhana/minio-go by harshavardhana 7 years ago
api: Failed call retry with region only when http.StatusBadRequest. Fixes #677 — committed to harshavardhana/minio-go by harshavardhana 7 years ago
api: Failed call retry with region only when http.StatusBadRequest. Fixes https://github.com/minio/minio-go/issues/677 — committed to restic/restic by harshavardhana 7 years ago
api: Failed call retry with region only when http.StatusBadRequest. Fixes #677 — committed to harshavardhana/minio-go by harshavardhana 7 years ago
api: Failed call retry with region only when http.StatusBadRequest. (#678) Fixes #677 — committed to minio/minio-go by harshavardhana 7 years ago

Most upvoted comments

We now have CI tests running against Minio and AWS S3, awesome! 😃

fd0 on May 13, 2017