milvus: [Bug]: The query performance of databases with small amounts of data is poor
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: v2.2.3
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.2.3
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: CPU(s):32, Mem:64G
- GPU: No
- Others: Three node k8s cluster
Current Behavior
I have two Milvus databases, one with only 2000 pieces of data and the other with a data volume of 2 million.
The sdk used is pymilvus, when the loaded database is 2 million, the QPS of the query performance tested is about 12700 req/s. However, when the loaded database is 2000, the QPS of the tested query is less than 700 req/s. I can ensure that all hardware environments are consistent, only the databases are different.
According to common sense, databases with less data theoretically have better query performance because they require less computation. But according to my testing results, the query performance of databases with low data volume is very, very, very poor! The resource utilization rate of the server is very low!
I don’t know if this is a bug or if the cluster configuration is incorrect.
Expected Behavior
The correct situation should be that the query performance of a database with less data volume is greater than that of a database with 2 million data volumes.
The explanation is:
- label_ test_ 128 databases only have 2000 pieces of data, random_ 128 database has 2 million pieces of data.
- Only the database is different, and other conditions remain consistent.
The source code is as follows:
import time
import numpy as np
from pymilvus import (
Collection,
connections,
)
from tqdm import tqdm
# 连接参数
# --------snip--------
MILVUS_HOST = "192.168.1.1"
MILVUS_PORT = 30090
data_dim = 128 # dim of vectors
top_k = 1 # 召回的数量
ef = 64
collection_name = "label_test_128" # 这个数据库里面只有2000条数据，有200万条数据的数据库为：random_128
partition_tag = ""
embedding_name = "embedding" # vector field name
expression = None
search_parameters = {
"anns_field": embedding_name,
"metric_type": "L2",
"params": {
"dim": data_dim,
"ef": ef,
"sp_value": ef
},
"limit": top_k,
"expression": expression,
}
# --------snip--------
class RecallByMilvus:
"""
专门用来做 milvus 向量检索的类
"""
def __init__(self):
connections.connect("default", host=MILVUS_HOST, port=MILVUS_PORT)
self.collection = None
def get_collection(self, collection_name):
try:
self.collection = Collection(collection_name)
except Exception as e:
print("Milvus create collection error: {}".format(e))
def search(self, vectors, embedding_name, collection_name, partition_names=None, output_fields=None):
"""
milvus 查询
:param vectors:
:param embedding_name:
:param collection_name:
:param partition_names:
:param output_fields:
:return:
"""
if output_fields is None or output_fields == "":
output_fields = []
else:
if not isinstance(output_fields, list):
output_fields = [output_fields]
if partition_names is None or partition_names == "":
partition_names = None
else:
if not isinstance(partition_names, list):
partition_names = [partition_names]
try:
self.get_collection(collection_name)
# 执行搜索
result = self.collection.search(
data=vectors,
anns_field=embedding_name,
# search_params,
param=search_parameters,
limit=top_k,
expr=None,
partition_names=partition_names,
output_fields=output_fields,
)
return result
except Exception as e:
print("Milvus recall error: {}".format(e))
def search_in_milvus(text_embedding, partition_names=None, output_fields=None):
"""
在 milvus 数据库里面搜索，分批进行搜索，最大的阈值为：16384
:param text_embedding: 需要搜索的向量
:param partition_names: 分区
:param output_fields: 输出的字段名
:return: 返回 pd 数据格式的 dict
"""
if not isinstance(text_embedding, np.ndarray) or text_embedding.size == 0:
return {}
return_dict = {}
distance_list = []
# 数组长度
vectors_len = text_embedding.shape[0]
# 分批处理大量的搜索
# 一批次搜索的最长长度
batch_size = 16384
# 可以共执行多少次搜索
range_len = int(vectors_len / batch_size) + 1
# 分批搜索
for index in tqdm(range(range_len)):
# index从0开始
if index * batch_size + batch_size < vectors_len:
batch_search_vectors = text_embedding[index * batch_size: index * batch_size + batch_size]
else:
# 到底了
batch_search_vectors = text_embedding[index * batch_size:]
# 执行搜索操作
result = recall_client.search(
vectors=batch_search_vectors,
embedding_name=embedding_name,
collection_name=collection_name,
partition_names=partition_names,
output_fields=output_fields,
)
# 如果有搜索结果
if result:
# 初始化 output field 的 key
for key in output_fields:
return_dict[key] = []
# 多个query的搜索结果
for hits in result:
# 召回的数据（根据 top_k 来控制的）
for hit in hits:
distance_list.append(hit.distance)
for key in output_fields:
return_dict[key].append(str(hit.entity.get(key)))
return_dict['distance'] = distance_list
# 返回符合 pd 数据格式的 dict
return return_dict
if __name__ == '__main__':
recall_client = RecallByMilvus()
# 测试搜索的性能
text_embedding = np.load("./test_search_18w.npy") # 这个是存放了18万条向量数据的文件，专门用来做查询的性能测试的
print("所有需要查询的向量的数量：{}".format(text_embedding.shape[0]))
search_time_start = time.time()
_dict = search_in_milvus(text_embedding=text_embedding, partition_names=partition_tag, output_fields=['id', 'text'])
print('都完成了，开始计算QPS...')
search_time_end = time.time()
search_count = text_embedding.shape[0]
total_cost = search_time_end - search_time_start
print("搜索 {} 条数据共花费时间：{} 秒".format(text_embedding.shape[0], total_cost))
print('QPS = {}'.format(search_count / total_cost))
print(_dict.keys())
Steps To Reproduce
Use the Python script above for QPS testing...
The QPS test output for a database with a data volume of 2 million is:
====================================================
所有需要查询的向量的数量：187488
100%|██████████| 12/12 [00:14<00:00, 1.21s/it]
都完成了，开始计算QPS...
搜索 187488 条数据共花费时间：14.556503057479858 秒
QPS = 12880.016530045606
dict_keys(['id', 'age', 'distance'])
====================================================
The QPS test output for a database with a data volume of 2000 is:
====================================================
所有需要查询的向量的数量：187488
100%|██████████| 12/12 [07:07<00:00, 35.60s/it]
都完成了，开始计算QPS...
搜索 187488 条数据共花费时间：427.26788091659546 秒
QPS = 438.8066793080533
dict_keys(['id', 'text', 'distance'])
====================================================
Milvus Log
Use the docker stats command to observe the resource usage of the Milvus cluster as follows:
3-node milvus cluster
- When the query data volume is 2 million
node1：
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
65226433d536 k8s_querynode_my-release-milvus-querynode-6c6744f796-xh5fs_default_15babedf-d561-4316-9951-3f7d0699bbb4_0 403.77% 2.042GiB / 62.65GiB 3.26% 0B / 0B 3.63MB / 4.1kB 44
0533ba56eb28 k8s_POD_my-release-milvus-querynode-6c6744f796-xh5fs_default_15babedf-d561-4316-9951-3f7d0699bbb4_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
586f40dabe3b k8s_nfs-client-provisioner_nfs-client-provisioner-59c95b5b7b-d9bnn_default_7e686304-7259-41af-9585-21c9b9bad2bb_6 0.18% 15.34MiB / 62.65GiB 0.02% 0B / 0B 16.9MB / 0B 67
d6460a8a0796 k8s_coredns_coredns-6d8c4cb4d-hvvh9_kube-system_f5fc5148-f03f-4ea4-aec3-c699de0d3c82_7 0.18% 28.02MiB / 170MiB 16.48% 0B / 0B 18.8MB / 0B 35
1b4f16aff88e k8s_coredns_coredns-6d8c4cb4d-n2jwd_kube-system_5efba5dd-1534-41b0-9b28-9408783823cb_6 0.20% 26.64MiB / 170MiB 15.67% 0B / 0B 19.2MB / 0B 37
f128c2fc26cc k8s_POD_nfs-client-provisioner-59c95b5b7b-d9bnn_default_7e686304-7259-41af-9585-21c9b9bad2bb_15 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
a18c54be4489 k8s_POD_coredns-6d8c4cb4d-hvvh9_kube-system_f5fc5148-f03f-4ea4-aec3-c699de0d3c82_17 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
bfdc27c8ecb2 k8s_POD_coredns-6d8c4cb4d-n2jwd_kube-system_5efba5dd-1534-41b0-9b28-9408783823cb_15 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
86bdc800126c k8s_kube-flannel_kube-flannel-ds-hnznn_kube-flannel_7d2b2d50-d96d-4a4e-8cc3-c344acd64daf_8 0.00% 27.41MiB / 62.65GiB 0.04% 0B / 0B 32.3MB / 0B 22
cb70f34fca14 k8s_POD_kube-flannel-ds-hnznn_kube-flannel_7d2b2d50-d96d-4a4e-8cc3-c344acd64daf_6 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
47ea0699fcd4 k8s_kube-proxy_kube-proxy-hk6vd_kube-system_fed9765c-da70-4f74-999a-3b34e42a1f54_7 0.56% 31.5MiB / 62.65GiB 0.05% 0B / 0B 43.6MB / 12.3kB 33
acc84ecc7f89 k8s_POD_kube-proxy-hk6vd_kube-system_fed9765c-da70-4f74-999a-3b34e42a1f54_7 0.00% 36KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
c004db602907 k8s_kube-scheduler_kube-scheduler-test202_kube-system_792e7d0f7c468c93275ba5541fbbc3f4_22 1.00% 36.48MiB / 62.65GiB 0.06% 0B / 0B 43MB / 0B 37
088a0a1f9a74 k8s_kube-apiserver_kube-apiserver-test202_kube-system_bc92d639771cf9790a31e2adaae41d9f_6 5.84% 523.2MiB / 62.65GiB 0.82% 0B / 0B 90.3MB / 0B 46
e0d6fde69ed8 k8s_etcd_etcd-test202_kube-system_538b2999e009f84440f8493e9e4a3298_7 1.55% 353.5MiB / 62.65GiB 0.55% 0B / 0B 644MB / 21.3GB 39
8023d634a76d k8s_kube-controller-manager_kube-controller-manager-test202_kube-system_ea2a8614266ce5fed190d888724ecbbe_21 3.20% 84.01MiB / 62.65GiB 0.13% 0B / 0B 79.2MB / 0B 43
a5f3513f1436 k8s_POD_kube-scheduler-test202_kube-system_792e7d0f7c468c93275ba5541fbbc3f4_7 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
3d450c35e6ae k8s_POD_kube-controller-manager-test202_kube-system_ea2a8614266ce5fed190d888724ecbbe_6 0.00% 36KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
0cb6830d7b7b k8s_POD_kube-apiserver-test202_kube-system_bc92d639771cf9790a31e2adaae41d9f_6 0.00% 112KiB / 62.65GiB 0.00% 0B / 0B 127kB / 0B 1
b57581ae50b4 k8s_POD_etcd-test202_kube-system_538b2999e009f84440f8493e9e4a3298_7 0.00% 192KiB / 62.65GiB 0.00% 0B / 0B 352kB / 0B 1
98a5dbdb84e0 attu_milvus 0.03% 72.26MiB / 62.65GiB 0.11% 6.43MB / 8.85MB 62.5MB / 0B 22
271db8cfa5c7 kube_dashboard 0.00% 38MiB / 62.65GiB 0.06% 163kB / 3.08MB 61MB / 0B 48
node2:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4173d39601a6 k8s_datacoord_my-release-milvus-datacoord-bf9f7d967-spcfd_default_ac1a630a-3503-4cd9-b733-547824ddfa12_0 1.50% 180.8MiB / 62.65GiB 0.28% 0B / 0B 1.72MB / 4.1kB 38
cdf985fb1a43 k8s_indexcoord_my-release-milvus-indexcoord-67df86f549-xcvzk_default_7d145761-d863-45d1-a479-af726b894583_0 0.10% 177.7MiB / 62.65GiB 0.28% 0B / 0B 1.5MB / 4.1kB 37
f23e60e17325 k8s_querynode_my-release-milvus-querynode-6c6744f796-vbvlq_default_f305ac32-f45d-4031-b27e-7609062c96c7_0 390.16% 2.033GiB / 62.65GiB 3.25% 0B / 0B 2.83MB / 4.1kB 44
3a6a629bb09e k8s_rootcoord_my-release-milvus-rootcoord-65689cb775-tsh8n_default_e30d93e7-79fc-473b-9652-aa2a3b9d75a7_0 3.05% 316.7MiB / 62.65GiB 0.49% 0B / 0B 1.73MB / 4.1kB 39
26d28a6bcf49 k8s_POD_my-release-milvus-rootcoord-65689cb775-tsh8n_default_e30d93e7-79fc-473b-9652-aa2a3b9d75a7_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
43f31d7d1506 k8s_POD_my-release-milvus-querynode-6c6744f796-vbvlq_default_f305ac32-f45d-4031-b27e-7609062c96c7_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
9f434acdaf1a k8s_POD_my-release-milvus-indexcoord-67df86f549-xcvzk_default_7d145761-d863-45d1-a479-af726b894583_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
a1bbb97d1944 k8s_POD_my-release-milvus-datacoord-bf9f7d967-spcfd_default_ac1a630a-3503-4cd9-b733-547824ddfa12_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
76dab3026915 k8s_cert-manager_cert-manager-webhook-7d4b5d8484-dtqqx_cert-manager_210bbc28-58e2-4019-bdd9-7f5289ca38ed_1 0.07% 28.78MiB / 62.65GiB 0.04% 0B / 0B 31.2MB / 0B 37
8c55dea66da5 k8s_POD_metrics-server-596bd64dd-94g2h_kube-system_178919d2-b8fa-4b67-8ab1-386ac8f880c8_62 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
e36c5b873a3b k8s_POD_cert-manager-webhook-7d4b5d8484-dtqqx_cert-manager_210bbc28-58e2-4019-bdd9-7f5289ca38ed_30 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
ec60204a2120 k8s_kube-flannel_kube-flannel-ds-wpwwr_kube-flannel_cf19db8c-729c-4fe5-9491-d88cf720cbc4_4 0.00% 26.41MiB / 62.65GiB 0.04% 0B / 0B 31.9MB / 0B 21
94b3bf70ef57 k8s_kube-proxy_kube-proxy-szsq4_kube-system_31060d95-b704-4d08-9775-c130994959ef_3 0.00% 27.55MiB / 62.65GiB 0.04% 0B / 0B 43.3MB / 12.3kB 31
33b0f15d3928 k8s_POD_kube-flannel-ds-wpwwr_kube-flannel_cf19db8c-729c-4fe5-9491-d88cf720cbc4_1 0.00% 36KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
f16c919727a6 k8s_POD_kube-proxy-szsq4_kube-system_31060d95-b704-4d08-9775-c130994959ef_1 0.00% 264KiB / 62.65GiB 0.00% 0B / 0B 610kB / 0B 1
node3:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
78c7b808d0d7 k8s_indexnode_my-release-milvus-indexnode-dcbfbfbbb-gjmxm_default_bb46a88c-6207-483a-9307-c6cc4981ba70_0 0.03% 172.5MiB / 251.4GiB 0.07% 0B / 0B 0B / 4.1kB 38
c29c09ca7ab3 k8s_datanode_my-release-milvus-datanode-7db5bb68d6-bgpd9_default_11df310a-8f14-4ef3-8bf3-b8103e37d9b6_0 1.83% 374MiB / 251.4GiB 0.15% 0B / 0B 4.28MB / 4.1kB 43
dc47a349d1c9 k8s_querynode_my-release-milvus-querynode-6c6744f796-vh2s6_default_c637c3d3-6296-4dbd-b19b-d7ef21fb6e6b_0 328.87% 2.019GiB / 251.4GiB 0.80% 0B / 0B 2.39MB / 4.1kB 47
77a2c5e1f593 k8s_querycoord_my-release-milvus-querycoord-6659d9745c-jdnlv_default_74d0086d-40de-4db0-925a-12eadcab2534_0 0.65% 211.4MiB / 251.4GiB 0.08% 0B / 0B 618kB / 4.1kB 39
f68e178517cb k8s_proxy_my-release-milvus-proxy-57745fd994-rzdjh_default_b05debf1-46a5-4e17-bea6-21bbca3d41e7_0 7.20% 346.7MiB / 251.4GiB 0.13% 0B / 0B 1.37MB / 4.1kB 42
3f8608649a91 k8s_POD_my-release-milvus-indexnode-dcbfbfbbb-gjmxm_default_bb46a88c-6207-483a-9307-c6cc4981ba70_0 0.00% 36KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
5f071708f26b k8s_POD_my-release-milvus-querynode-6c6744f796-vh2s6_default_c637c3d3-6296-4dbd-b19b-d7ef21fb6e6b_0 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
f06005164872 k8s_POD_my-release-milvus-datanode-7db5bb68d6-bgpd9_default_11df310a-8f14-4ef3-8bf3-b8103e37d9b6_0 0.00% 44KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
b9b7c2424e83 k8s_POD_my-release-milvus-querycoord-6659d9745c-jdnlv_default_74d0086d-40de-4db0-925a-12eadcab2534_0 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
4e1bf49fda06 k8s_POD_my-release-milvus-proxy-57745fd994-rzdjh_default_b05debf1-46a5-4e17-bea6-21bbca3d41e7_0 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
d5cda56e35b2 k8s_manager_milvus-operator-5b754b5fbf-h4wfq_milvus-operator_b5cd6906-f921-4f66-8b79-608350dbd7c2_1 0.02% 46.65MiB / 1GiB 4.56% 0B / 0B 51MB / 0B 38
545cf0487335 k8s_cert-manager_cert-manager-cainjector-76f7798c9-phrnh_cert-manager_1e03d1d8-6aa8-4d19-9eab-23ce559146da_1 0.04% 67.84MiB / 251.4GiB 0.03% 0B / 0B 28.4MB / 0B 37
11d8e492c403 k8s_cert-manager_cert-manager-6888d6b69b-2djhq_cert-manager_11694f31-ea89-4753-b5ba-9e5c7511cab5_1 0.02% 36.56MiB / 251.4GiB 0.01% 0B / 0B 45.4MB / 0B 37
d017eeb406f0 k8s_POD_milvus-operator-5b754b5fbf-h4wfq_milvus-operator_b5cd6906-f921-4f66-8b79-608350dbd7c2_30 0.00% 36KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
99895703f2ad k8s_POD_cert-manager-cainjector-76f7798c9-phrnh_cert-manager_1e03d1d8-6aa8-4d19-9eab-23ce559146da_29 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
caea06274d9f k8s_POD_cert-manager-6888d6b69b-2djhq_cert-manager_11694f31-ea89-4753-b5ba-9e5c7511cab5_32 0.00% 44KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
da0f15bb6992 k8s_kube-flannel_kube-flannel-ds-ht7xx_kube-flannel_780486d2-e1e1-4753-9678-ebf43e12f9b0_6 0.00% 28.96MiB / 251.4GiB 0.01% 0B / 0B 3.32MB / 0B 20
06fcd54e79a1 k8s_kube-proxy_kube-proxy-69zb2_kube-system_acc44ada-8768-4319-84fb-1f82ad911feb_3 0.00% 27.46MiB / 251.4GiB 0.01% 0B / 0B 43.7MB / 12.3kB 32
276a5eb08379 k8s_POD_kube-proxy-69zb2_kube-system_acc44ada-8768-4319-84fb-1f82ad911feb_2 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
208ce46f3f75 k8s_POD_kube-flannel-ds-ht7xx_kube-flannel_780486d2-e1e1-4753-9678-ebf43e12f9b0_2 0.00% 288KiB / 251.4GiB 0.00% 0B / 0B 618kB / 0B 1
dbda0e73376a pulsar_milvus 11.74% 3.207GiB / 251.4GiB 1.28% 10.2GB / 14.6GB 2.27GB / 26.6GB 1101
- When the query data volume is 2000
node1:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
65226433d536 k8s_querynode_my-release-milvus-querynode-6c6744f796-xh5fs_default_15babedf-d561-4316-9951-3f7d0699bbb4_0 103.34% 2.065GiB / 62.65GiB 3.30% 0B / 0B 3.63MB / 4.1kB 44
0533ba56eb28 k8s_POD_my-release-milvus-querynode-6c6744f796-xh5fs_default_15babedf-d561-4316-9951-3f7d0699bbb4_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
586f40dabe3b k8s_nfs-client-provisioner_nfs-client-provisioner-59c95b5b7b-d9bnn_default_7e686304-7259-41af-9585-21c9b9bad2bb_6 0.38% 15.43MiB / 62.65GiB 0.02% 0B / 0B 16.9MB / 0B 67
d6460a8a0796 k8s_coredns_coredns-6d8c4cb4d-hvvh9_kube-system_f5fc5148-f03f-4ea4-aec3-c699de0d3c82_7 0.13% 28.31MiB / 170MiB 16.65% 0B / 0B 18.8MB / 0B 35
1b4f16aff88e k8s_coredns_coredns-6d8c4cb4d-n2jwd_kube-system_5efba5dd-1534-41b0-9b28-9408783823cb_6 0.19% 27.1MiB / 170MiB 15.94% 0B / 0B 19.2MB / 0B 37
f128c2fc26cc k8s_POD_nfs-client-provisioner-59c95b5b7b-d9bnn_default_7e686304-7259-41af-9585-21c9b9bad2bb_15 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
a18c54be4489 k8s_POD_coredns-6d8c4cb4d-hvvh9_kube-system_f5fc5148-f03f-4ea4-aec3-c699de0d3c82_17 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
bfdc27c8ecb2 k8s_POD_coredns-6d8c4cb4d-n2jwd_kube-system_5efba5dd-1534-41b0-9b28-9408783823cb_15 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
86bdc800126c k8s_kube-flannel_kube-flannel-ds-hnznn_kube-flannel_7d2b2d50-d96d-4a4e-8cc3-c344acd64daf_8 1.16% 27.16MiB / 62.65GiB 0.04% 0B / 0B 32.3MB / 0B 22
cb70f34fca14 k8s_POD_kube-flannel-ds-hnznn_kube-flannel_7d2b2d50-d96d-4a4e-8cc3-c344acd64daf_6 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
47ea0699fcd4 k8s_kube-proxy_kube-proxy-hk6vd_kube-system_fed9765c-da70-4f74-999a-3b34e42a1f54_7 0.00% 31.48MiB / 62.65GiB 0.05% 0B / 0B 43.6MB / 12.3kB 33
acc84ecc7f89 k8s_POD_kube-proxy-hk6vd_kube-system_fed9765c-da70-4f74-999a-3b34e42a1f54_7 0.00% 36KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
c004db602907 k8s_kube-scheduler_kube-scheduler-test202_kube-system_792e7d0f7c468c93275ba5541fbbc3f4_22 0.49% 35.86MiB / 62.65GiB 0.06% 0B / 0B 43MB / 0B 37
088a0a1f9a74 k8s_kube-apiserver_kube-apiserver-test202_kube-system_bc92d639771cf9790a31e2adaae41d9f_6 15.51% 522MiB / 62.65GiB 0.81% 0B / 0B 90.3MB / 0B 46
e0d6fde69ed8 k8s_etcd_etcd-test202_kube-system_538b2999e009f84440f8493e9e4a3298_7 6.38% 356.6MiB / 62.65GiB 0.56% 0B / 0B 644MB / 21.3GB 39
8023d634a76d k8s_kube-controller-manager_kube-controller-manager-test202_kube-system_ea2a8614266ce5fed190d888724ecbbe_21 4.32% 83.2MiB / 62.65GiB 0.13% 0B / 0B 79.2MB / 0B 43
a5f3513f1436 k8s_POD_kube-scheduler-test202_kube-system_792e7d0f7c468c93275ba5541fbbc3f4_7 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
3d450c35e6ae k8s_POD_kube-controller-manager-test202_kube-system_ea2a8614266ce5fed190d888724ecbbe_6 0.00% 36KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
0cb6830d7b7b k8s_POD_kube-apiserver-test202_kube-system_bc92d639771cf9790a31e2adaae41d9f_6 0.00% 112KiB / 62.65GiB 0.00% 0B / 0B 127kB / 0B 1
b57581ae50b4 k8s_POD_etcd-test202_kube-system_538b2999e009f84440f8493e9e4a3298_7 0.00% 192KiB / 62.65GiB 0.00% 0B / 0B 352kB / 0B 1
98a5dbdb84e0 attu_milvus 0.02% 72.47MiB / 62.65GiB 0.11% 6.44MB / 8.85MB 62.5MB / 0B 22
271db8cfa5c7 kube_dashboard 0.00% 38MiB / 62.65GiB 0.06% 163kB / 3.08MB 61MB / 0B 48
node2:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4173d39601a6 k8s_datacoord_my-release-milvus-datacoord-bf9f7d967-spcfd_default_ac1a630a-3503-4cd9-b733-547824ddfa12_0 1.71% 180.9MiB / 62.65GiB 0.28% 0B / 0B 1.72MB / 4.1kB 38
cdf985fb1a43 k8s_indexcoord_my-release-milvus-indexcoord-67df86f549-xcvzk_default_7d145761-d863-45d1-a479-af726b894583_0 0.52% 177.7MiB / 62.65GiB 0.28% 0B / 0B 1.5MB / 4.1kB 37
f23e60e17325 k8s_querynode_my-release-milvus-querynode-6c6744f796-vbvlq_default_f305ac32-f45d-4031-b27e-7609062c96c7_0 2.31% 1.958GiB / 62.65GiB 3.13% 0B / 0B 2.83MB / 4.1kB 44
3a6a629bb09e k8s_rootcoord_my-release-milvus-rootcoord-65689cb775-tsh8n_default_e30d93e7-79fc-473b-9652-aa2a3b9d75a7_0 3.48% 316.7MiB / 62.65GiB 0.49% 0B / 0B 1.73MB / 4.1kB 39
26d28a6bcf49 k8s_POD_my-release-milvus-rootcoord-65689cb775-tsh8n_default_e30d93e7-79fc-473b-9652-aa2a3b9d75a7_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
43f31d7d1506 k8s_POD_my-release-milvus-querynode-6c6744f796-vbvlq_default_f305ac32-f45d-4031-b27e-7609062c96c7_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
9f434acdaf1a k8s_POD_my-release-milvus-indexcoord-67df86f549-xcvzk_default_7d145761-d863-45d1-a479-af726b894583_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
a1bbb97d1944 k8s_POD_my-release-milvus-datacoord-bf9f7d967-spcfd_default_ac1a630a-3503-4cd9-b733-547824ddfa12_0 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
76dab3026915 k8s_cert-manager_cert-manager-webhook-7d4b5d8484-dtqqx_cert-manager_210bbc28-58e2-4019-bdd9-7f5289ca38ed_1 2.53% 28.96MiB / 62.65GiB 0.05% 0B / 0B 31.2MB / 0B 37
8c55dea66da5 k8s_POD_metrics-server-596bd64dd-94g2h_kube-system_178919d2-b8fa-4b67-8ab1-386ac8f880c8_62 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
e36c5b873a3b k8s_POD_cert-manager-webhook-7d4b5d8484-dtqqx_cert-manager_210bbc28-58e2-4019-bdd9-7f5289ca38ed_30 0.00% 40KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
ec60204a2120 k8s_kube-flannel_kube-flannel-ds-wpwwr_kube-flannel_cf19db8c-729c-4fe5-9491-d88cf720cbc4_4 0.00% 26.27MiB / 62.65GiB 0.04% 0B / 0B 31.9MB / 0B 21
94b3bf70ef57 k8s_kube-proxy_kube-proxy-szsq4_kube-system_31060d95-b704-4d08-9775-c130994959ef_3 0.00% 27.69MiB / 62.65GiB 0.04% 0B / 0B 43.3MB / 12.3kB 31
33b0f15d3928 k8s_POD_kube-flannel-ds-wpwwr_kube-flannel_cf19db8c-729c-4fe5-9491-d88cf720cbc4_1 0.00% 36KiB / 62.65GiB 0.00% 0B / 0B 0B / 0B 1
f16c919727a6 k8s_POD_kube-proxy-szsq4_kube-system_31060d95-b704-4d08-9775-c130994959ef_1 0.00% 264KiB / 62.65GiB 0.00% 0B / 0B 610kB / 0B 1
node3:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
78c7b808d0d7 k8s_indexnode_my-release-milvus-indexnode-dcbfbfbbb-gjmxm_default_bb46a88c-6207-483a-9307-c6cc4981ba70_0 0.03% 172.5MiB / 251.4GiB 0.07% 0B / 0B 0B / 4.1kB 38
c29c09ca7ab3 k8s_datanode_my-release-milvus-datanode-7db5bb68d6-bgpd9_default_11df310a-8f14-4ef3-8bf3-b8103e37d9b6_0 3.69% 374MiB / 251.4GiB 0.15% 0B / 0B 4.28MB / 4.1kB 43
dc47a349d1c9 k8s_querynode_my-release-milvus-querynode-6c6744f796-vh2s6_default_c637c3d3-6296-4dbd-b19b-d7ef21fb6e6b_0 101.17% 1.991GiB / 251.4GiB 0.79% 0B / 0B 2.39MB / 4.1kB 47
77a2c5e1f593 k8s_querycoord_my-release-milvus-querycoord-6659d9745c-jdnlv_default_74d0086d-40de-4db0-925a-12eadcab2534_0 0.39% 216.8MiB / 251.4GiB 0.08% 0B / 0B 618kB / 4.1kB 39
f68e178517cb k8s_proxy_my-release-milvus-proxy-57745fd994-rzdjh_default_b05debf1-46a5-4e17-bea6-21bbca3d41e7_0 0.45% 296.9MiB / 251.4GiB 0.12% 0B / 0B 1.37MB / 4.1kB 42
3f8608649a91 k8s_POD_my-release-milvus-indexnode-dcbfbfbbb-gjmxm_default_bb46a88c-6207-483a-9307-c6cc4981ba70_0 0.00% 36KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
5f071708f26b k8s_POD_my-release-milvus-querynode-6c6744f796-vh2s6_default_c637c3d3-6296-4dbd-b19b-d7ef21fb6e6b_0 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
f06005164872 k8s_POD_my-release-milvus-datanode-7db5bb68d6-bgpd9_default_11df310a-8f14-4ef3-8bf3-b8103e37d9b6_0 0.00% 44KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
b9b7c2424e83 k8s_POD_my-release-milvus-querycoord-6659d9745c-jdnlv_default_74d0086d-40de-4db0-925a-12eadcab2534_0 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
4e1bf49fda06 k8s_POD_my-release-milvus-proxy-57745fd994-rzdjh_default_b05debf1-46a5-4e17-bea6-21bbca3d41e7_0 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
d5cda56e35b2 k8s_manager_milvus-operator-5b754b5fbf-h4wfq_milvus-operator_b5cd6906-f921-4f66-8b79-608350dbd7c2_1 0.27% 46.78MiB / 1GiB 4.57% 0B / 0B 51MB / 0B 38
545cf0487335 k8s_cert-manager_cert-manager-cainjector-76f7798c9-phrnh_cert-manager_1e03d1d8-6aa8-4d19-9eab-23ce559146da_1 0.20% 67.84MiB / 251.4GiB 0.03% 0B / 0B 28.4MB / 0B 37
11d8e492c403 k8s_cert-manager_cert-manager-6888d6b69b-2djhq_cert-manager_11694f31-ea89-4753-b5ba-9e5c7511cab5_1 0.01% 36.56MiB / 251.4GiB 0.01% 0B / 0B 45.4MB / 0B 37
d017eeb406f0 k8s_POD_milvus-operator-5b754b5fbf-h4wfq_milvus-operator_b5cd6906-f921-4f66-8b79-608350dbd7c2_30 0.00% 36KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
99895703f2ad k8s_POD_cert-manager-cainjector-76f7798c9-phrnh_cert-manager_1e03d1d8-6aa8-4d19-9eab-23ce559146da_29 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
caea06274d9f k8s_POD_cert-manager-6888d6b69b-2djhq_cert-manager_11694f31-ea89-4753-b5ba-9e5c7511cab5_32 0.00% 44KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
da0f15bb6992 k8s_kube-flannel_kube-flannel-ds-ht7xx_kube-flannel_780486d2-e1e1-4753-9678-ebf43e12f9b0_6 0.00% 28.97MiB / 251.4GiB 0.01% 0B / 0B 3.32MB / 0B 20
06fcd54e79a1 k8s_kube-proxy_kube-proxy-69zb2_kube-system_acc44ada-8768-4319-84fb-1f82ad911feb_3 0.01% 27.46MiB / 251.4GiB 0.01% 0B / 0B 43.7MB / 12.3kB 32
276a5eb08379 k8s_POD_kube-proxy-69zb2_kube-system_acc44ada-8768-4319-84fb-1f82ad911feb_2 0.00% 40KiB / 251.4GiB 0.00% 0B / 0B 0B / 0B 1
208ce46f3f75 k8s_POD_kube-flannel-ds-ht7xx_kube-flannel_780486d2-e1e1-4753-9678-ebf43e12f9b0_2 0.00% 288KiB / 251.4GiB 0.00% 0B / 0B 618kB / 0B 1
dbda0e73376a pulsar_milvus 11.30% 3.207GiB / 251.4GiB 1.28% 10.2GB / 14.7GB 2.27GB / 26.6GB 1101
Anything else?
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17 (12 by maintainers)
@Lunpopo 2000 pieces of data are too few. Milvus’s data is segmented, with a default size of 512M per segment. The index will only be built after the segments are sealed. Your collection with 2M pieces of data has already had its index built, which is why the QPS can meet expectations. However, with only 2000 pieces of data, it does not meet the requirements for building an index and will use brute force search instead, resulting in a low QPS.