milvus: [Bug]: [chaos][standalone]Milvus hangs at creating index after standalone pod kill chaos deleted

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: `91be4b1`
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2):2.0.0rc8.dev23
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Milvus hangs at search after standalone pod kill chaos deleted

Expected Behavior

All operations work well

Steps To Reproduce

1. cd `tests/python_client/chaos`

2. modify param value and run script `chaos_test.sh`
pod="standalone"
chaos_type="pod_kill"

A common way to reproduce
deploy a standalone milvus by helm 
use `kubectl delete` to kill standalone pod multi times (about 5~10times in 1 min)
run hello_milvus.py to check the results


Anything else?

standalone.log

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 48 (47 by maintainers)

Commits related to this issue

Most upvoted comments

IndexCoord may assign task to a dead IndexNode and it will retry many times. @czs007 is fixing it.