milvus: [Bug]: Investigate the Cpu Usage on small instance when the system is idle

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

https://github.com/milvus-io/milvus/issues/22571 https://github.com/milvus-io/milvus/issues/17942

We will see many similar issues that said cpu is high after data insertion.

I have some wild guess about it, but we need to verify on how to reproduce:

  1. Single collection with enough data -> Create a 2c8G Standalone and insert 8GB memory into it, do some search and leave it for at least 1 day, see the data growth and cpu usage(should be less than 0.5 core)
  2. Multi collection -> Create a 2c8G Standalone and create 100 collections, see what will happen
  3. Multi collection - >Create a 2c8G Standalone and create 100 collections, each collection insert 10000 entities and trigger index build

Expected Behavior

CPU usage is below 1 CPU

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 31 (21 by maintainers)

Most upvoted comments

@elonzh v2.2.12 was released this week. please try to run your tests if convenient. thanks.

I have no idea about the increase in CPU usage. I stopped the server when received the alert.

But I am sure there are no requests when this issue happens because it is a test server for evaluation.

hi @elonzh , I know what’s going on. It will be fixed in 2.2.6, thanks for your feedback.

  1. After 3h of testing, the cpu resource usage was reduced to less than 0.5, using about 0.05c. image

  2. create 100 collections ,cpu usage 0.15, mem usage : 482m image image

  3. Conclusion: cpu use below 0.5c image

image