milvus: [Bug]: Milvus v2.2.12 fails to create collection on S3 bucket

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.2.12
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: Intel i5/16GB
- GPU: None
- Others:

Current Behavior

Trying to configure S3. After S3 configure, it connects successfully to Milvus, but fails to create the collection, Collection() just hang. Nothing was created on my S3 bucket.

Expected Behavior

Collection() to success and files to appear on the S3 bucket.

Steps To Reproduce

Here are my settings:
milvus.yaml

minio:
  address: rgw.ourdomain.com
  port: 443
  accessKeyID: "<my_s3_access_key_id>"
  secretAccessKey: "<my_s3_secret_access_key>"
  useSSL: true
  bucketName: "<my_bucket_name>"
  rootPath: files
  useIAM: false
  cloudProvider: aws
  iamEndpoint: ""
  logLevel: debug


cat docker-compose.yml

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ./milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    volumes:
      - ./milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.2.12
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
    volumes:
      - ./milvus.yaml:/milvus/configs/milvus.yaml
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus


Here is how tried:

import sys

from pymilvus import connections
from pymilvus import CollectionSchema, FieldSchema, DataType
from pymilvus import Collection

try:
    connections.connect(alias="default", host="192.168.1.110", port="19530")
    print(f"Connect to Milvus success.")
except Exception as e:
    print(f"Connect to Milvus failed!")
    print("Error:", str(e))
    sys.exit(0)

pk = FieldSchema(
  name="pk",
  dtype=DataType.INT64,
  is_primary=True,
  auto_id=True,
)

text = FieldSchema(
  name="text",
  dtype=DataType.VARCHAR,
  max_length=8192,
)

vector = FieldSchema(
  name="vector",
  dtype=DataType.FLOAT_VECTOR,
  dim=1536
)

schema = CollectionSchema(
  fields=[pk, text, vector],
  description="News",
  enable_dynamic_field=True
)

collection_name = "news3"

try:
    collection = Collection(
	    name=collection_name,
	    schema=schema,
	    using='default',
	    shards_num=2
	    )
    print(f"Create Collection success.")
except Exception as e:
    print(f"Create Collection failed!")
    print("Error:", str(e))
    connections.disconnect("default")
    sys.exit(0)

Milvus Log

[2023/07/27 10:14:24.045 +00:00] [INFO] [storage/minio_chunk_manager.go:137] [“minio chunk manager init success.”] [bucketname=<my_bucket_name>] [root=files] [2023/07/27 10:14:24.045 +00:00] [INFO] [querynode/query_node.go:325] [“queryNode try to connect etcd success”] [MetaRootPath=by-dev/meta] [2023/07/27 10:14:24.045 +00:00] [INFO] [querynode/segment_loader.go:949] [“SegmentLoader created”] [ioPoolSize=32] [cpuPoolSize=4] [2023/07/27 10:14:24.047 +00:00] [INFO] [storage/minio_chunk_manager.go:137] [“minio chunk manager init success.”] [bucketname=<my_bucket_name>] [root=files] [2023/07/27 10:14:24.048 +00:00] [INFO] [storage/minio_chunk_manager.go:137] [“minio chunk manager init success.”] [bucketname=<my_bucket_name>] [root=files] [2023/07/27 10:14:24.048 +00:00] [INFO] [indexcoord/index_coord.go:263] [“IndexCoord new minio chunkManager success”] [2023/07/27 10:14:24.048 +00:00] [INFO] [indexcoord/flush_segment_watcher.go:87] [“flushSegmentWatcher reloadFromKV”] [2023/07/27 10:14:24.049 +00:00] [INFO] [indexcoord/flush_segment_watcher.go:109] [“flushSegmentWatcher reloadFromKV success”] [etcdRevision=997637] [2023/07/27 10:14:24.049 +00:00] [INFO] [indexcoord/index_coord.go:277] [“IndexCoord new task scheduler success”] [2023/07/27 10:14:24.049 +00:00] [INFO] [indexcoord/index_coord.go:280] [“IndexCoord init finished”] [2023/07/27 10:14:24.049 +00:00] [INFO] [sessionutil/session_util.go:201] [“Session try to connect to etcd”] [2023/07/27 10:14:24.049 +00:00] [INFO] [sessionutil/session_util.go:216] [“Session connect to etcd success”] [2023/07/27 10:14:24.049 +00:00] [INFO] [indexcoord/service.go:148] [“IndexCoord try to wait for RootCoord ready”] [2023/07/27 10:14:24.050 +00:00] [INFO] [sessionutil/session_util.go:525] ["SessionUtil GetSessions "] [prefix=rootcoord] [key=rootcoord] [address=192.168.128.4:53100] [2023/07/27 10:14:24.050 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=192.168.128.4:53100] [2023/07/27 10:14:24.051 +00:00] [INFO] [datacoord/channel_store.go:165] [“channel store reload channel”] [nodeID=144] [channel=by-dev-rootcoord-dml_0_442755433698213276v0] [2023/07/27 10:14:24.051 +00:00] [INFO] [datacoord/channel_store.go:165] [“channel store reload channel”] [nodeID=144] [channel=by-dev-rootcoord-dml_12_442755433702915009v0] [2023/07/27 10:14:24.051 +00:00] [INFO] [datacoord/channel_store.go:165] [“channel store reload channel”] [nodeID=144] [channel=by-dev-rootcoord-dml_13_442755433702915009v1] [2023/07/27 10:14:24.051 +00:00] [INFO] [datacoord/channel_store.go:165] [“channel store reload channel”] [nodeID=144] [channel=by-dev-rootcoord-dml_1_442755433698213276v1] [2023/07/27 10:14:24.051 +00:00] [INFO] [datacoord/channel_store.go:165] [“channel store reload channel”] [nodeID=144] [channel=by-dev-rootcoord-dml_2_443077785304966170v0] [2023/07/27 10:14:24.051 +00:00] [INFO] [datacoord/channel_store.go:165] [“channel store reload channel”] [nodeID=144] [channel=by-dev-rootcoord-dml_3_443077785304966170v1] [2023/07/27 10:14:24.052 +00:00] [INFO] [sessionutil/session_util.go:554] [“SessionUtil GetSessions “] [prefix=datanode] [key=datanode-144] [address=192.168.128.4:21124] [2023/07/27 10:14:24.052 +00:00] [INFO] [datacoord/server.go:443] [“DataCoord success to get DataNode sessions”] [sessions=”{"datanode-144":{"ServerID":144,"ServerName":"datanode","Address":"192.168.128.4:21124","TriggerKill":true,"Version":"2.2.12"}}”] [2023/07/27 10:14:24.052 +00:00] [INFO] [datacoord/channel_manager.go:210] [“processing watch info”] [“watch state”=ReleaseSuccess] [“channel name”=by-dev-rootcoord-dml_0_442755433698213276v0] [2023/07/27 10:14:24.054 +00:00] [INFO] [sessionutil/session_util.go:525] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=192.168.128.4:53100] [2023/07/27 10:14:24.054 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=192.168.128.4:53100] [2023/07/27 10:14:24.090 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #132 SubChannel #133] grpc: addrConn.createTransport failed to connect to {\n "Addr": "192.168.128.4:53100",\n "ServerName": "192.168.128.4:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 192.168.128.4:53100: i/o timeout"”]

[2023/07/27 10:14:24.090 +00:00] [ERROR] [grpcclient/client.go:353] [“ClientBase ReCall grpc second call get error”] [role=rootcoord] [error=“err: failed to connect 192.168.128.4:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:67 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:91 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:106 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:165 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:99 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:353\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:67\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:91\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:106\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:165\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:99\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[…].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120”]

[2023/07/27 10:14:45.764 +00:00] [INFO] [rootcoord/meta_table.go:468] [“remove collection”] [dbID=1] [name=news3] [id=443141261209707638] [aliases=“[]”]

Anything else?

No response