milvus: [Bug]: [chaos]Flush hangs at e2e test after deleting pulsar pod kill chaos
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version: master-20211225-132b3c2
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): 2.0.0rc9.dev24
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
Flush hangs at e2e test after deleting datacoord or rootcoord pod kill chaos
Expected Behavior
all operations work well after deleting chaos
Steps To Reproduce
# reproduce on local k8s env
cd `tests/python_client/chaos`
modify milvus tag in `cluster-values.yaml` to `master-20211225-132b3c2`
run command `bash chaos_test.sh rootcoord pod_kill`
# run GitHub action
https://github.com/milvus-io/milvus/actions/workflows/pod-kill-chaos-test.yaml
Anything else?
datacoord log: https://github.com/milvus-io/milvus/suites/4764821710/artifacts/132476087
rootcoord log: https://github.com/milvus-io/milvus/suites/4764821710/artifacts/132476097
log path is k8s_logs/chaos_test
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 43 (42 by maintainers)
Commits related to this issue
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able to retry ... — committed to XuanYang-cn/milvus by XuanYang-cn 3 years ago
- Add flowgraph manager and event manager (#15102) 1. Add flowgraph manager to manager start and release of dataSyncService 2. Add event manager to manager etcd watchInfo event - Make put event able ... — committed to milvus-io/milvus by XuanYang-cn 2 years ago
@XuanYang-cn Would you help to figure out the reason why it cost so long? 👍