milvus: [Bug]: [benchmark][cluster]Milvus search failed , datanode crash and client side reported: "InactiveRpcError of RPC that terminated"

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.2.0-20230309-130ab6da
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):2.3.0.dev45
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

case: test_concurrent_locust_100m_hnsw_ddl_dql_filter_kafka_cluster argo task : fouramf-concurrent-wlp9h

server:

fouramf-concurrent-wlp9h-1-etcd-0                                 1/1     Running            0                 15h     10.104.4.53     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-etcd-1                                 1/1     Running            0                 15h     10.104.9.71     4am-node14   <none>           <none>
fouramf-concurrent-wlp9h-1-etcd-2                                 1/1     Running            0                 15h     10.104.5.65     4am-node12   <none>           <none>
fouramf-concurrent-wlp9h-1-kafka-0                                1/1     Running            2 (15h ago)       15h     10.104.4.52     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-kafka-1                                1/1     Running            1 (15h ago)       15h     10.104.9.69     4am-node14   <none>           <none>
fouramf-concurrent-wlp9h-1-kafka-2                                1/1     Running            2 (15h ago)       15h     10.104.5.63     4am-node12   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-datacoord-77477c57dd-fpgn4      1/1     Running            15 (5m48s ago)    15h     10.104.4.40     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-datanode-85c9c4466c-pk8sk       0/1     CrashLoopBackOff   117 (4m54s ago)   15h     10.104.4.42     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-indexcoord-559fc86544-tt6mg     1/1     Running            0                 15h     10.104.4.43     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-indexnode-7954bd777-kkw94       1/1     Running            0                 15h     10.104.4.45     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-proxy-67f9f4f678-d8gwf          1/1     Running            0                 15h     10.104.4.39     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-querycoord-6799b7547-r5fr9      1/1     Running            0                 15h     10.104.4.44     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-querynode-97c86b8bf-gg8k6       1/1     Running            0                 15h     10.104.4.41     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-querynode-97c86b8bf-qqmvh       1/1     Running            0                 15h     10.104.5.55     4am-node12   <none>           <none>
fouramf-concurrent-wlp9h-1-milvus-rootcoord-6f9ddcf9f-2wxhl       1/1     Running            0                 15h     10.104.4.38     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-minio-0                                1/1     Running            0                 15h     10.104.4.51     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-minio-1                                1/1     Running            0                 15h     10.104.9.64     4am-node14   <none>           <none>
fouramf-concurrent-wlp9h-1-minio-2                                1/1     Running            0                 15h     10.104.5.58     4am-node12   <none>           <none>
fouramf-concurrent-wlp9h-1-minio-3                                1/1     Running            0                 15h     10.104.1.119    4am-node10   <none>           <none>
fouramf-concurrent-wlp9h-1-zookeeper-0                            1/1     Running            0                 15h     10.104.4.50     4am-node11   <none>           <none>
fouramf-concurrent-wlp9h-1-zookeeper-1                            1/1     Running            0                 15h     10.104.9.68     4am-node14   <none>           <none>
fouramf-concurrent-wlp9h-1-zookeeper-2                            1/1     Running            0                 15h     10.104.5.64     4am-node12   <none>           <none>

client error log:

Expected Behavior

No response

Steps To Reproduce

1. create a collection or use an existing collection
        2. build index on vector column
        3. insert a certain number of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. build index on on scalars column or not
        7. count the total number of rows
        8. load collection
        9. perform concurrent operations
        10. clean all collections or not

Milvus Log

No response

Anything else?

No response