milvus: [Bug]: querynode: Cannot create producer on topic with backlog quota exceeded

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.0 rc8
- Deployment mode(standalone or cluster): cluster
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

time=“2021-11-24T03:05:20Z” level=info msg=“[Reconnecting to broker in 1m3.495358951s]” producerID=4 producer_name=standalone-0-319 topic=“persistent://public/default/test-searchResult-0” time=“2021-11-24T03:06:23Z” level=error msg=“[Failed to create producer]” error=“server error: ProducerBlockedQuotaExceededError: Cannot create producer on topic with backlog quota exceeded” producerID=4 producer_name=standalone-0-319 topic=“persistent://public/default/test-searchResult-0”

Expected Behavior

No response

Steps To Reproduce

when I create multiple milvus-proxy pod, just one proxy pod is active and the cluster running fine, then I shutdown the active one, another proxy is active, but this time, the search request from pymilvus will hang.
I look at the querynode log, the search assignment is finished successfully, but when write it to pulsar, the error above occur.

Anything else?

look at #12207