milvus: [Bug]: Collection not getting loaded, also loading progress is shown as 0, "no flush channel found for the segment, unable to flush", "Failed to get shard delegator, channel not found"

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.3.3
- Deployment mode(standalone or cluster): Milvus cluster deployed in kubernetes
- MQ type(rocksmq, pulsar or kafka): External Kafka (AWS MSK)   
- SDK version(e.g. pymilvus v2.0.0rc2): Pymilvus 2.3.3
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

After ingesting around 1.7 billion entities across 100 collections, when am trying to do collection.load(), am getting collection not loaded exception and loading progress is shown as 0 as well

These are key errors i see in data coords and data nodes “no flush channel found for the segment, unable to flush”, “Failed to get shard delegator, channel not found”

Expected Behavior

collection getting loaded successfully and also search operation happening successfully

i tried with single collection and 31 million entities ingested, its working as expected

but its not working with 100 collection and 1.7 billion entities

Steps To Reproduce

No response

Milvus Log

milvus-log.tar.gz

Anything else?

Here

milvusdiskann1 -> cluster is used for 1.7 billion entities and 100 collections, where collection is not getting loaded and thus search is not happening successfully

milvusdiskannsmall -> cluster is used for 31 million entities and 1 collection, where collection is getting loaded successfully and also search is happening successfully