milvus: [Bug]: [chaos][standalone]Milvus hangs at creating index after standalone pod kill chaos deleted
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: `91be4b1`
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2):2.0.0rc8.dev23
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
Milvus hangs at search after standalone pod kill chaos deleted
Expected Behavior
All operations work well
Steps To Reproduce
1. cd `tests/python_client/chaos`
2. modify param value and run script `chaos_test.sh`
pod="standalone"
chaos_type="pod_kill"
A common way to reproduce
deploy a standalone milvus by helm
use `kubectl delete` to kill standalone pod multi times (about 5~10times in 1 min)
run hello_milvus.py to check the results
Anything else?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 48 (47 by maintainers)
Commits related to this issue
- Fix issue #11275 issue: #11275 Signed-off-by: sunby <bingyi.sun@zilliz.com> — committed to sunby/milvus by deleted user 3 years ago
- Fix issue #11275 issue: #11275 Signed-off-by: sunby <bingyi.sun@zilliz.com> — committed to sunby/milvus by deleted user 3 years ago
- Fix issue #11275 issue: #11275 Signed-off-by: sunby <bingyi.sun@zilliz.com> — committed to sunby/milvus by deleted user 3 years ago
- Improve proxy manager in rootcoord issue: #11275 Signed-off-by: sunby <bingyi.sun@zilliz.com> — committed to sunby/milvus by deleted user 3 years ago
- Improve proxy manager in rootcoord (#12303) issue: #11275 Signed-off-by: sunby <bingyi.sun@zilliz.com> Co-authored-by: sunby <bingyi.sun@zilliz.com> — committed to milvus-io/milvus by sunby 3 years ago
IndexCoord may assign task to a dead IndexNode and it will retry many times. @czs007 is fixing it.