milvus: [Bug]: Proxy not healthy on Docker Swarm

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.2.10
- Deployment mode(standalone or cluster): standalone on docker swarm
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When I run this docker-compose with docker compose up -d it works fine. but when I run it with docker stack deploy -c docker-compose.yml milvus it’s not working. first it gives “Proxy not healthy” then after about 10-15 minutes it works fine.

` version: “3.9”

services:

etcd: image: quay.io/coreos/etcd:v3.5.5 networks: - frs-public environment: - ETCD_AUTO_COMPACTION_MODE=revision - ETCD_AUTO_COMPACTION_RETENTION=1000 - ETCD_QUOTA_BACKEND_BYTES=4294967296 - ETCD_SNAPSHOT_COUNT=50000 volumes: - type: bind source: /home/frs/milvus/etcd target: /etcd command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd deploy: placement: constraints: - “node.role==manager”

minio: image: minio/minio:RELEASE.2023-03-20T20-16-18Z networks: - frs-public volumes: - type: bind source: /home/frs/milvus/minio target: /minio_data environment: MINIO_ACCESS_KEY: minioadmin MINIO_SECRET_KEY: minioadmin command: minio server /minio_data healthcheck: test: [“CMD”, “curl”, “-f”, “http://localhost:9000/minio/health/live”] interval: 30s timeout: 20s retries: 3 deploy: placement: constraints: - “node.role==manager”

standalone: image: milvusdb/milvus:v2.2.10 networks: - frs-public command: [“milvus”, “run”, “standalone”] environment: ETCD_ENDPOINTS: etcd:2379 MINIO_ADDRESS: minio:9000 volumes: - type: bind source: /home/frs/milvus/milvus target: /var/lib/milvus ports: - “19530:19530” - “9091:9091” depends_on: - etcd - minio deploy: placement: constraints: - “node.role==manager”

networks: frs-public: `

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

Complete log: https://file.io/vxEe999zcntN

[2023/06/20 09:10:12.537 +00:00] [WARN] [rootcoord/proxy_client_manager.go:239] [“proxy client is empty, GetMetrics will not send to any client”] [2023/06/20 09:10:12.538 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] [“len of msess”=0] [2023/06/20 09:10:12.538 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available datacoord, check datacoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:12.538 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=datacoord] [error=“err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [2023/06/20 09:10:12.538 +00:00] [WARN] [client/client.go:94] [“QueryCoordClient msess key not existed”] [key=querycoord] [2023/06/20 09:10:12.538 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available querycoord, check querycoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:12.539 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=querycoord] [error=“err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [2023/06/20 09:10:12.539 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] [“len of msess”=0] [2023/06/20 09:10:12.539 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available datacoord, check datacoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=datacoord] [error=“err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:12.540 +00:00] [WARN] [client/client.go:94] [“QueryCoordClient msess key not existed”] [key=querycoord] [2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available querycoord, check querycoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:12.540 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=querycoord] [error=“err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:12.540 +00:00] [WARN] [rootcoord/quota_center.go:143] [“quotaCenter sync metrics failed”] [error=“err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [2023/06/20 09:10:13.083 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.084 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:13.093 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.093 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:13.284 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #753 SubChannel #754] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:13.284 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run\n”] [2023/06/20 09:10:13.285 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.285 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:13.294 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974 github.com/milvus-io/milvus/internal/datacoord.(*Server).hasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n”] [2023/06/20 09:10:13.294 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #755 SubChannel #756] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled"”] [2023/06/20 09:10:13.295 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:13.295 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:13.486 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #757 SubChannel #758] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:13.486 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:148\ngithub.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexcoord/service.go:81\ngithub.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[…].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120”] [2023/06/20 09:10:13.496 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #759 SubChannel #760] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:13.496 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974 github.com/milvus-io/milvus/internal/datacoord.(*Server).hasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396 github.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163 github.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:234\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).hasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:974\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection.func1\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:361\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:360\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:396\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).unwatchDroppedChannels\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:246\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:163\ngithub.com/milvus-io/milvus/internal/datacoord.(*Cluster).Startup\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/cluster.go:57\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initServiceDiscovery\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:441\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:310\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:283\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:241\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[…].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120”] [2023/06/20 09:10:14.070 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.070 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:14.164 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.164 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:14.172 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.172 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:14.271 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #761 SubChannel #762] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:14.271 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n”] [2023/06/20 09:10:14.272 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.272 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:14.365 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #763 SubChannel #764] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:14.366 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n”] [2023/06/20 09:10:14.367 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.367 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:14.372 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52 github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n”] [2023/06/20 09:10:14.372 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #765 SubChannel #766] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled"”] [2023/06/20 09:10:14.373 +00:00] [INFO] [sessionutil/session_util.go:477] [“SessionUtil GetSessions “] [prefix=rootcoord] [key=rootcoord] [address=10.0.0.61:53100] [2023/06/20 09:10:14.373 +00:00] [INFO] [client/client.go:109] [“RootCoordClient GetSessions success”] [address=10.0.0.61:53100] [2023/06/20 09:10:14.472 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #767 SubChannel #768] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: operation was canceled"”] [2023/06/20 09:10:14.472 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51 github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:387\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:295\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[…].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120”] [2023/06/20 09:10:14.568 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #769 SubChannel #770] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:14.568 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52 github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:164\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:98\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[…].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120”] [2023/06/20 09:10:14.574 +00:00] [WARN] [grpclog/grpclog.go:46] [”[core][Channel #771 SubChannel #772] grpc: addrConn.createTransport failed to connect to {\n "Addr": "10.0.0.61:53100",\n "ServerName": "10.0.0.61:53100",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n}. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.61:53100: i/o timeout"”] [2023/06/20 09:10:14.574 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=rootcoord] [error=“err: failed to connect 10.0.0.61:53100, reason: context deadline exceeded\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52 github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:130\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:65\ngithub.com/milvus-io/milvus/internal/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/internal/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:89\ngithub.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/funcutil/func.go:104\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:275\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:172\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[…].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:120”] [2023/06/20 09:10:15.537 +00:00] [WARN] [rootcoord/proxy_client_manager.go:239] [“proxy client is empty, GetMetrics will not send to any client”] [2023/06/20 09:10:15.539 +00:00] [WARN] [client/client.go:94] [“QueryCoordClient msess key not existed”] [key=querycoord] [2023/06/20 09:10:15.539 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] [“len of msess”=0] [2023/06/20 09:10:15.539 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available querycoord, check querycoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:15.539 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available datacoord, check datacoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:312\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:15.539 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=querycoord] [error=“err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [2023/06/20 09:10:15.539 +00:00] [WARN] [grpcclient/client.go:318] [“ClientBase ReCall grpc first call get error”] [role=datacoord] [error=“err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:317 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [2023/06/20 09:10:15.540 +00:00] [WARN] [client/client.go:98] ["DataCoordClient, not existed in msess "] [key=datacoord] [“len of msess”=0] [2023/06/20 09:10:15.540 +00:00] [WARN] [client/client.go:94] [“QueryCoordClient msess key not existed”] [key=querycoord] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available datacoord, check datacoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:158] [“failed to get client address”] [error=“find no available querycoord, check querycoord state”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).connect\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:158\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).GetGrpcClient\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:131\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).callOnce\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:256\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=datacoord] [error=“err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:15.540 +00:00] [ERROR] [grpcclient/client.go:330] [“ClientBase ReCall grpc second call get error”] [role=querycoord] [error=“err: find no available querycoord, check querycoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”] [stack=“github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:330\ngithub.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/client/client.go:359\ngithub.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func1\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:188\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75”] [2023/06/20 09:10:15.540 +00:00] [WARN] [rootcoord/quota_center.go:143] [“quotaCenter sync metrics failed”] [error=“err: find no available datacoord, check datacoord state\n, /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:329 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[…]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:527 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/go/src/github.com/milvus-io/milvus/internal/rootcoord/quota_center.go:213 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).syncMetrics.func2\n/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit\n”]

Anything else?

No response