milvus: [Bug]: [Nightly] Load balance failed reporting 'failed to balance segments, err=TaskStale'

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:5f02f09
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.0.dev30
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Load balance failed reporting ‘failed to balance segments, err=TaskStale’.

[2022-09-22T18:41:41.366Z] [2022-09-22 18:10:29 - ERROR - pymilvus.decorators]: RPC error: [load_balance], <MilvusException: (code=1, message=failed to balance segments, err=TaskStale)>, <Time:{'RPC start': '2022-09-22 18:10:29.127591', 'RPC error': '2022-09-22 18:10:29.612304'}> (decorators.py:100)
[2022-09-22T18:41:41.366Z] [2022-09-22 18:10:29 - ERROR - ci_test]: Traceback (most recent call last):
[2022-09-22T18:41:41.366Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2022-09-22T18:41:41.366Z]     res = func(*args, **_kwargs)
[2022-09-22T18:41:41.366Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2022-09-22T18:41:41.366Z]     return func(*arg, **kwargs)
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/utility.py", line 543, in load_balance
[2022-09-22T18:41:41.366Z]     return _get_connection(using).load_balance(collection_name, src_node_id, dst_node_ids, sealed_segment_ids, timeout=timeout)
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 101, in handler
[2022-09-22T18:41:41.366Z]     raise e
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 97, in handler
[2022-09-22T18:41:41.366Z]     return func(*args, **kwargs)
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 127, in handler
[2022-09-22T18:41:41.366Z]     ret = func(self, *args, **kwargs)
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 79, in handler
[2022-09-22T18:41:41.366Z]     raise e
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2022-09-22T18:41:41.366Z]     return func(self, *args, **kwargs)
[2022-09-22T18:41:41.366Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1042, in load_balance
[2022-09-22T18:41:41.366Z]     raise MilvusException(status.error_code, status.reason)
[2022-09-22T18:41:41.366Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to balance segments, err=TaskStale)>
[2022-09-22T18:41:41.366Z]  (api_request.py:39)
[2022-09-22T18:41:41.366Z] [2022-09-22 18:10:29 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=failed to balance segments, err=TaskStale)> (api_request.py:40)

Expected Behavior

PASSED

Steps To Reproduce

https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/114/pipeline/143/

Milvus Log

1. milvus log: artifacts-milvus-distributed-kafka-nightly-114-pymilvus-e2e-logs.tar.gz
2. collection name: utility_Wk9P7ZE4
3. failed timeline: [2022-09-22T18:10:36.576Z] [gw0] [ 99%] FAILED testcases/test_utility.py::TestUtilityAdvanced::test_load_balance_normal

Anything else?

No response