seaweedfs: New version of seaweedfs can't list directory using s3 api

@chrislusf @kmlebedev

Potentially connected to PR #4778 It seems that the functionality for listing entries with the method ListDirectoryPrefixedEntries may not have been implemented for Redis and Mongodb, which could be a potential root cause.

Listing subdirectories is done recursively, the ListDirectoryPrefixedEntries method is not required.

https://github.com/seaweedfs/seaweedfs/blob/8908810376e671b34262295f1ad558ca43db58c2/weed/s3api/s3api_objects_list_handlers.go#L379

I review the code and found ListDirectoryPrefixedEntries method is required by doListFilerEntries -> then goes to https://github.com/seaweedfs/seaweedfs/blob/f24c7e803f0a14d35e2bb40a2ed838c80b8cf2e8/weed/s3api/s3api_objects_list_handlers.go#L334-L334 -> then https://github.com/seaweedfs/seaweedfs/blob/f24c7e803f0a14d35e2bb40a2ed838c80b8cf2e8/weed/server/filer_grpc_server.go#L40 https://github.com/seaweedfs/seaweedfs/blob/f24c7e803f0a14d35e2bb40a2ed838c80b8cf2e8/weed/server/filer_grpc_server.go#L59 … -> at the end it call https://github.com/seaweedfs/seaweedfs/blob/f24c7e803f0a14d35e2bb40a2ed838c80b8cf2e8/weed/filer/filer.go#L350-L352 which call ListDirectoryPrefixedEntries method

@chrislusf @kmlebedev After code review and debug，I found the root cause：

ListDirectoryPrefixedEntries in weed/filer/filerstore_wrapper.go called.

and it will try to use ListDirectoryPrefixedEntries of actualStore to get the file list

and when ListDirectoryPrefixedEntries return ErrUnsupportedListDirectoryPrefixed ERROR

it will fall back to ListDirectoryEntries with https://github.com/seaweedfs/seaweedfs/blob/117fbba5bf1947a280376e688d24b3a9b1c589c9/weed/filer/filerstore_wrapper.go#L277-L320

it will works well in the past , but unfortunately，with https://github.com/seaweedfs/seaweedfs/pull/4391 the parameter “limit” set to 1 so it will only worked well when the directory you list is the first of the parent directory.

restic init create only one directory named keys and it worked well, after we restic backup directory “data” and other directory created by restic. so s3api cannot list keys directory , it report the key error

I am not familiar with golang , but if I thought change the limit parameter in the fall back in https://github.com/seaweedfs/seaweedfs/blob/117fbba5bf1947a280376e688d24b3a9b1c589c9/weed/filer/filerstore_wrapper.go#L271-L273 will fix this bug and not need to full fill every ListDirectoryPrefixedEntries in every kind of filer store.

I found the same problem . s3api cannot list object with prefix when filer store is Cassandra is there any progress on this issue？

Hey, I was testing SeaweedFS with our cross-platform S3 GUI (https://s3drive.app/) and can confirm that problem is present in 3.55 and is not present in 3.45.

This creates: explicitFolder/ key:

3.55 (INCORRECT)

3.45 (CORRECT)

The implicitFolder (folder without explicit key/entry ending with /, but instead virtual CommonPrefix derived from all of the files and sub-folders inside of that folder) works fine on both versions:

Alternative way - using AWS CLI

Insert key

aws --endpoint=http://localhost:8333/ s3api put-object --bucket seabucket --key explicit/

Query

aws --endpoint=http://localhost:8333/ s3api list-objects-v2 --bucket seabucket --prefix explicit --delimiter "/"

Response 3.55 (INCORRECT)

{
    "Contents": [
        {
            "Key": "explicit/",
            "LastModified": "2023-08-08T11:22:57+00:00",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e-0\"",
            "Size": 0,
            "StorageClass": "STANDARD",
            "Owner": {
                "ID": "3e8"
            }
        }
    ]
}

Response 3.45 (CORRECT)

{
    "CommonPrefixes": [
        {
            "Prefix": "explicit/"
        }
    ]
}

S3 Filer CMD

I wasn’t touching master or volume processes, I was only switching version of S3 Filer.

# v3.55
weed filer -s3 -ip=127.0.0.1 -master=127.0.0.1:9333
# v3.45
weed345 filer -s3 -ip=127.0.0.1 -master=127.0.0.1:9333

CORS missing

You would be able to try it out using our S3 browser client (https://web.s3drive.app) but it I wasn’t able to setup CORS properly for SeaweedFS without running front NginX.

Hello,

@chrislusf have you been able to reproduce it yet?

I can confirm this happened with the release of 3.46. Directory listings using the s3 service works as expected through version 3.45.

It is not clear to me which of these two PRs might have caused the issue #4390 or #4391