milvus: [Bug]: Search failed with error `reason=target node id not match target id = 3, node id = 12` after pulsar pod kill chaos test

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: master-20221206-f8cff798
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus==2.3.0.dev15
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2022-12-06T22:16:38.598Z] [2022-12-06 22:16:29 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 120], kwargs: {} (api_request.py:56)

[2022-12-06T22:16:38.598Z] [2022-12-06 22:16:29 - DEBUG - ci_test]: (api_response) : None  (api_request.py:31)

[2022-12-06T22:16:38.598Z] [2022-12-06 22:16:29 - INFO - ci_test]: [test][2022-12-06T22:16:29Z] [0.00333510s] DeleteChecker__huJONjuF load -> None (wrapper.py:30)

[2022-12-06T22:16:38.598Z] [2022-12-06 22:16:29 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.01373642304506658, 0.09407661457501135, 0.037831905386391126, 0.028200136389675192, 0.1333814968391419, 0.11025818933976621, 0.10980963426000147, 0.13031918810532903, 0.03308945619420152, 0.11283760831918727, 0.023766451770019223, 0.019642186799281227, 0.12117970130462996, 0.06948829826502975, ......, kwargs: {} (api_request.py:56)

[2022-12-06T22:16:38.598Z] [2022-12-06 22:16:29 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=target node id not match target id = 3, node id = 12)>, <Time:{'RPC start': '2022-12-06 22:16:29.318095', 'RPC error': '2022-12-06 22:16:29.439993'}> (decorators.py:108)

[2022-12-06T22:16:38.598Z] [2022-12-06 22:16:29 - ERROR - ci_test]: Traceback (most recent call last):

[2022-12-06T22:16:38.598Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper

[2022-12-06T22:16:38.598Z]     res = func(*args, **_kwargs)

[2022-12-06T22:16:38.598Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request

[2022-12-06T22:16:38.598Z]     return func(*arg, **kwargs)

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 610, in search

[2022-12-06T22:16:38.598Z]     res = conn.search(self._name, data, anns_field, param, limit, expr,

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler

[2022-12-06T22:16:38.598Z]     raise e

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 105, in handler

[2022-12-06T22:16:38.598Z]     return func(*args, **kwargs)

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2022-12-06T22:16:38.598Z]     ret = func(self, *args, **kwargs)

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 85, in handler

[2022-12-06T22:16:38.598Z]     raise e

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler

[2022-12-06T22:16:38.598Z]     return func(self, *args, **kwargs)

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 469, in search

[2022-12-06T22:16:38.598Z]     return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, auto_id=auto_id, **kwargs)

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 438, in _execute_search_requests

[2022-12-06T22:16:38.598Z]     raise pre_err

[2022-12-06T22:16:38.598Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 429, in _execute_search_requests

[2022-12-06T22:16:38.598Z]     raise MilvusException(response.status.error_code, response.status.reason)

[2022-12-06T22:16:38.599Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=target node id not match target id = 3, node id = 12)>

[2022-12-06T22:16:38.599Z]  (api_request.py:39)

[2022-12-06T22:16:38.599Z] [2022-12-06 22:16:29 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=target node id not match target id = 3, node id = 12)> (api_request.py:40)