milvus: [Bug]: Flush hangs when running hello_milvus.py after pulsar recovered from pod kill chaos

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20220610-36ad9895
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus==2.1.0.dev69
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The action has timed out when running hello_milvus.py Outputting duplicate logs in proxy pod

[2022/06/12 18:57:51.851 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:52.355 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:52.356 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:52.861 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:52.866 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:53.370 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:53.370 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:53.873 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:53.877 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:54.379 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:54.381 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:54.884 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:54.885 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]

Expected Behavior

all test cases passed

Steps To Reproduce

see https://github.com/milvus-io/milvus/runs/6851761481?check_suite_focus=true

Milvus Log

failed job: https://github.com/milvus-io/milvus/runs/6851761481?check_suite_focus=true log: https://github.com/milvus-io/milvus/suites/6898913792/artifacts/267696164

Anything else?

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (21 by maintainers)

Most upvoted comments

It still happened in version master-20220613-e6225d92 Your PR is merged in the version

failed job: https://github.com/zhuwenxing/milvus/runs/6857375279?check_suite_focus=true but the log is not collected due to the job is timeout, not the failure

I will try it again