milvus: [Bug]: search latency increased 100% after upsert done

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: master-20221109-0dc659a0
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): 2.2.0dev72

Current Behavior

search latency increased 100%+ after upsert done

Expected Behavior

search latency shall keep stable, and not increase

Steps To Reproduce

1. create a collection with 5 million 128d vectors
2. create hnsw index and load
3. start a search client to keep search permanently 
4. start a new client to delete 1m entities and insert 1m entities
5. check the search latency

Milvus Log

pod names:

yanliang-cluster-1cu-milvus-datanode-848447979-rzsz5            1/1     Running     0               28m
yanliang-cluster-1cu-milvus-indexnode-997d5cc59-78spn           1/1     Running     0               28m
yanliang-cluster-1cu-milvus-mixcoord-58b7f4b757-lczmx           1/1     Running     0               28m
yanliang-cluster-1cu-milvus-proxy-85848fcb4d-58gt9              1/1     Running     0               28m
yanliang-cluster-1cu-milvus-querynode-6fc57668b9-6n2l2          1/1     Running     0               13m

Anything else?

after upsert, the segment count keeps 11, but there could be 1-2 empty(row count =0) segments in searching requests. Not sure whether this is the reason.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16 (16 by maintainers)

Commits related to this issue

Enable to gen compaction plan for stale segments See also: #20502 Signed-off-by: yangxuan <xuan.yang@zilliz.com> — committed to XuanYang-cn/milvus by XuanYang-cn 2 years ago
Enable to gen compaction plan for stale segments See also: #20502 Signed-off-by: yangxuan <xuan.yang@zilliz.com> — committed to XuanYang-cn/milvus by XuanYang-cn 2 years ago
Enable to gen compaction plan for stale segments (#20742) See also: #20502 Signed-off-by: yangxuan <xuan.yang@zilliz.com> Signed-off-by: yangxuan <xuan.yang@zilliz.com> — committed to milvus-io/milvus by XuanYang-cn 2 years ago
Enable to gen compaction plan for stale segments (#20741) See also: #20502 Signed-off-by: yangxuan <xuan.yang@zilliz.com> Signed-off-by: yangxuan <xuan.yang@zilliz.com> — committed to milvus-io/milvus by XuanYang-cn 2 years ago
Enable to gen compaction plan for stale segments (#20741) See also: #20502 Signed-off-by: yangxuan <xuan.yang@zilliz.com> Signed-off-by: yangxuan <xuan.yang@zilliz.com> — committed to yah01/milvus by XuanYang-cn 2 years ago

Most upvoted comments

manually call flush, the growing segments becomes index in a second, and the search latency keeps without any improvement. please check one of segments 437427639710798269 around 2022/11/17 11:30 UTC time. First it was growing, then it became indexed after i manually flush the collection

yanliang-cluster-1cu-milvus-datanode-758bd58dbf-th7tl           1/1     Running     0               22h     10.102.7.10     devops-node11   <none>           <none>
yanliang-cluster-1cu-milvus-indexnode-7d7c5567c7-qp2wt          1/1     Running     0               22h     10.102.5.103    devops-node21   <none>           <none>
yanliang-cluster-1cu-milvus-mixcoord-5bd98dd779-fj28n           1/1     Running     0               22h     10.102.5.104    devops-node21   <none>           <none>
yanliang-cluster-1cu-milvus-proxy-7755788f4b-bl7r9              1/1     Running     0               22h     10.102.9.28     devops-node13   <none>           <none>
yanliang-cluster-1cu-milvus-querynode-548cbd8498-v7l5h          1/1     Running     0               22h     10.102.8.124    devops-node12   <none>           <none>

yanliang567 on Nov 18, 2022