milvus: search timeout for a while

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.2.2
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): centos
- CPU/Memory: 6c32gb
- GPU: 
- Others:

Current Behavior

The search is timeout for 3 mins when my cluster reach the business peak period.

  1. Here is the Search Latency dashboard: image
  2. Here are parts of querynode metrics and Search Segment Latency is quite well: image
  3. Here is Search Group NQ dashboard which is shaked at the same time: image

This cluster is my online environment and I hope to providing a solution and reasons as soon as possible.Thanks a lot~

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

Uploading milvus.log.tar.gz… Uploading milvus.log.tar.gz…

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 24 (11 by maintainers)

Most upvoted comments

1689063686418 image image image image image image image image image image image

There are all metrics.FYI~