milvus: [Bug]: Milvus crashes if search with expr in a large id list

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: 2.0.0
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: 4 / 16GB
- GPU: 
- Others:

Current Behavior

I used Milvus 1.0.0 before and want to use 2.0.0. Honestly, while considering resource usage, I am disappointed with 2.0.0. I have a 160.000 vector that is 2048 float values. It must be 160.000 * 2048 * 8 bytes = 2.6 GB, but when I check the docker volume size, it seems 17 GB disk size. There is a big difference. RAM usage is also huge, it’s about 4-5 GB for 160K vectors.

I use Milvus 2.0.0 with default configurations, but Milvus 1.0.0 is better than 2.0.0 while considering resource usage. Do I miss something? Can anyone explain what the reason for the difference is?

Expected Behavior

Normal disk and ram usage

Steps To Reproduce

No response

Anything else?

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19 (15 by maintainers)

Most upvoted comments

@yanliang567, @xiaofan-luan, thanks for the answers. I wonder about how to decrease disk usage. I deleted some insert_log files thinking they were unnecessary logs, but the data was affected and could never be searched again. Then I learned that I shouldn’t do it again 😃 I think I can manipulate the milvus.yaml config file to remove log files for a short period. Which parameter is used to do it?

These are settings for message queue of milvus standalone :

rocksmq:
  path: /var/lib/milvus/rdb_data # The path where the message is stored in rocksmq
  rocksmqPageSize: 2147483648 # 2 GB, 2 * 1024 * 1024 * 1024 bytes, The size of each page of messages in rocksmq
  retentionTimeInMinutes: 10080 # 7 days, 7 * 24 * 60 minutes, The retention time of the message in rocksmq.
  retentionSizeInMB: 8192 # 8 GB, 8 * 1024 MB, The retention size of the message in rocksmq.

You can modify those retentionTimeInMinutes and retentionSizeInMB two options to decrease disk usage.

LoveEachDay on Feb 15, 2022