seaweedfs: 501 not implemented on GET.BUCKET

Happens for example when trying to use cockroachdb backup onto seaweedfs s3.

Audit log entry

2023-09-19T09:34:02+00:00	s3.access	{"request_id":"121940fcf63dff0c62972347c18bad2b","key":"/","bucket":"dt","status":501,"user_agent":"aws-sdk-go/1.40.37 (go1.19.10; linux; amd64)","remote_ip":"dt","operation":"REST.GET.BUCKET","host_id":"","host_header":"dt","time":1695116042,"error_code":"NotImplemented","signature_version":"SigV4","requester":"dt"}

Debug logs

I0919 11:40:41.824370 auth_credentials.go:347 auth error: 0
I0919 11:40:41.824395 filer_client.go:120 read directory: directory:"/buckets"  limit:2147483647
I0919 11:40:41.825613 error_handler.go:96 status 200 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><DisplayName></DisplayName><ID></ID></Owner><Buckets></Buckets></ListAllMyBucketsResult>
I0919 11:40:42.575906 auth_credentials.go:260 v4 auth type
I0919 11:40:42.576002 auth_credentials.go:289 user name: dt actions: [Read:dt Write:dt List:dt Tagging:dt Admin:dt], action: Read
I0919 11:40:42.576055 s3api_object_handlers.go:177 GetObjectHandler dt /2023/09/19-094042.45/BACKUP_MANIFEST
I0919 11:40:42.576077 s3api_object_handlers.go:357 s3 proxying GET to http://dtd/buckets/dt/2023/09/19-094042.45/BACKUP_MANIFEST
I0919 11:40:42.576992 error_handler.go:96 status 404 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/dt/2023/09/19-094042.45/BACKUP_MANIFEST</Resource><RequestId>1695116442576920827</RequestId><Key>2023/09/19-094042.45/BACKUP_MANIFEST</Key><BucketName>dt</BucketName></Error>
I0919 11:40:42.590864 auth_credentials.go:260 v4 auth type
I0919 11:40:42.591028 auth_credentials.go:289 user name: dt actions: [Read:dt Write:dt List:dt Tagging:dt Admin:dt], action: Read
I0919 11:40:42.591070 s3api_object_handlers.go:177 GetObjectHandler dt /2023/09/19-094042.45/BACKUP-LOCK-901270872132845569
I0919 11:40:42.591140 s3api_object_handlers.go:357 s3 proxying GET to http://dtd/buckets/dt/2023/09/19-094042.45/BACKUP-LOCK-901270872132845569
I0919 11:40:42.591822 error_handler.go:96 status 404 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/dt/2023/09/19-094042.45/BACKUP-LOCK-901270872132845569</Resource><RequestId>1695116442591743441</RequestId><Key>2023/09/19-094042.45/BACKUP-LOCK-901270872132845569</Key><BucketName>dt</BucketName></Error>
I0919 11:40:42.605766 auth_credentials.go:260 v4 auth type
I0919 11:40:42.605908 auth_credentials.go:289 user name: dt actions: [Read:dt Write:dt List:dt Tagging:dt Admin:dt], action: Read
I0919 11:40:42.605951 s3api_object_handlers.go:177 GetObjectHandler dt /2023/09/19-094042.45/BACKUP_MANIFEST
I0919 11:40:42.605972 s3api_object_handlers.go:357 s3 proxying GET to http://dtd/buckets/dt/2023/09/19-094042.45/BACKUP_MANIFEST
I0919 11:40:42.606594 error_handler.go:96 status 404 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Resource>/dt/2023/09/19-094042.45/BACKUP_MANIFEST</Resource><RequestId>1695116442606540234</RequestId><Key>2023/09/19-094042.45/BACKUP_MANIFEST</Key><BucketName>dt</BucketName></Error>
I0919 11:40:42.620524 auth_credentials.go:260 v4 auth type
I0919 11:40:42.620637 auth_credentials.go:289 user name: dt actions: [Read:dt Write:dt List:dt Tagging:dt Admin:dt], action: List
I0919 11:40:42.620670 s3api_objects_list_handlers.go:98 ListObjectsV1Handler dt
I0919 11:40:42.620740 error_handler.go:96 status 501 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Resource>/dt</Resource><RequestId>1695116442620683995</RequestId><BucketName>dt</BucketName></Error>

Seems to be this one

https://github.com/seaweedfs/seaweedfs/blob/0bb97709d41b1be4c74f01dcc65aac6d5f88bd16/weed/s3api/s3api_objects_list_handlers.go#L106-L109

Describe the bug

Get Actions should work, but don’t in this case.

System Setup

  • weed 3.56

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 17 (15 by maintainers)

Most upvoted comments

Having not checked this in detail, is cursor.maxKeys decreasing per iteration, or per path that matched? It looks to me that it is the former, but would need to be the latter. At least that is what I remember when I took a look at the logic.

Logic wise per se, your implementation is up to the spec yes.

ah yeah, from the docs:

Each rolled-up result counts as only one return against the MaxKeys value.

I was under the impression somehow that the roll up just found the 1000 keys and did the grouping that way. I’ll test that change and edit the patch. Thanks for catching

e: tested another cockroach backup and updated patch. i don’t have that many keys yet in the bucket to have run into this potentially causing an issue so ymmv