[ Subject of the issue ]

Description

Describe your issue in as much detail as possible here… We have 65 peers. 25 are boot nodes. 40 are validators. On a few of the boot nodes, we get this error. Our --max-peers set at 100

Your environment

• OS and version. Ubuntu 20.04
• The version of the Polygon Edge. 0.6.1 (Confirm the version of your Polygon edge client by running the following command: polygon-edge version --grpc-address GRPC_ADDRESS)
• The branch that causes this issue.
• Locally or Cloud hosted (which provider). AWS cloud
• Please confirm if the validators are running under containerized environment (K8s, Docker, etc.). no

Steps to reproduce

• Tell us how to reproduce this issue.
• Where the issue is, if you know.
• Which commands triggered the issue, if any.
• Provide us with the content of your genesis file.
• Provide us with commands that you used to start your validators.

polygon-edge server --max-peers 100 --data-dir ./newchain --chain genesis.json --libp2p 0.0.0.0:1478 --jsonrpc 0.0.0.0:10002 --price-limit 7150000000000000 --max-slots 40960000 --seal

• Provide us with the peer list of each of your validators by running the following command: polygon-edge peers list --grpc-address GRPC_ADDRESS.
• Is the chain producing blocks and serving customers atm? yes

Expected behavior

• Tell us what should happen. Run without errors
• Tell us what happened instead. We get the output below

Logs

Provide us with debug logs from all of your validators by setting logging to debug output with: server --log-level debug

-12-08T11:02:10.725Z [INFO] polygon.server.ibft.consensus: sequence started: height=144082 │·····································································································2022-12-08T11:02:10.725Z [INFO] polygon.server.ibft.consensus: round started: round=0 │·····································································································2022-12-08T11:02:11.206Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│·····································································································iu2HAm1Kr667HcnaRMHxNukSfSXgffiHXbj4gDuJXUbi2owYmv err=“peer rejected; insufficient capacity” │····································································································· 2022-12-08T11:02:11.206Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│·····································································································iu2HAmCyVHRmX9JHp83tHy7m1PzGUfkFGswXdH45NfYfzQhkdV err=“peer rejected; insufficient capacity” │····································································································· 2022-12-08T11:02:11.206Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│·····································································································iu2HAmJiGCv3zj3ayDufia5qrtLECgFuAGjT1atPSNKtiKteim err=“peer rejected; insufficient capacity” │·····································································································2022-12-08T11:02:11.206Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│····································································································· iu2HAmSAsAbuyiWMMB61ABo2srknccXZDvar7SPvn1NTnPFsA7 err=“peer rejected; insufficient capacity” │·····································································································2022-12-08T11:02:11.207Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│····································································································· iu2HAmKAAUkxTDQKVkRyzD5EyPNq97kC2XYMsHV4vi56wsCDVF err=“peer rejected; insufficient capacity” │·····································································································2022-12-08T11:02:11.207Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│····································································································· iu2HAmFMLK2coEznboxnt2tm4J2qe4L3BwGHHi5PG8WiRGf6RJ err=“peer rejected; insufficient capacity” │·····································································································2022-12-08T11:02:11.207Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│····································································································· iu2HAmRyXwWVjsevF5S4tmsStVcug7DJ4o9VXTMST7tBwb9UkK err=“peer rejected; insufficient capacity” │·····································································································2022-12-08T11:02:11.207Z [ERROR] polygon.network.discovery: Failed to add new peer to routing table: peer=16U│····································································································· iu2HAmSE9vH7EwYgmPgsVBV34zDXq2BWrxzBTHDpfjwStnvAYW err=“peer rejected; insufficient capacity” │····································································································· 2022-12-08T11:02:14.098Z [INFO] polygon.blockchain: new block: number=144082 txs=0 hash=0x275177d0618b2cd940│····································································································· 59e2198c89bfd7c1b72c58ffbfcf753e74e4edcbf22189 parent=0x3dd74b926d4891fcb9255647325c7141d892f2c24c4d5f2acf248│····································································································· 23ba539322d generation_time_in_seconds=3 │····································································································· 2022-12-08T11:02:14.423Z [INFO] polygon.blockchain: block already inserted: block=144082 source=consensus │····································································································· 2022-12-08T11:02:14.423Z [INFO] polygon.server.ibft: block committed: number=144082 hash=0x275177d0618b2cd94│····································································································· 059e2198c89bfd7c1b72c58ffbfcf753e74e4edcbf22189 validation_type=bls validators=65 committed=44 │····································································································· 2022-12-08T11:02:14.423Z [INFO] polygon.server.ibft.consensus: sequence done: height=144082 │····································································································· 2022-12-08T11:02:14.423Z [INFO] polygon.server.ibft: canceled sequence: sequence=144082 │····································································································· 2022-12-08T11:02:14.423Z [INFO] polygon.server.ibft.consensus: sequence

Proposed solution

If you have an idea on how to fix this issue, please write it down here, so we can begin discussing it