mev-boost: Proposal missing

Hello there. Running Geth v1.11.6, Prysm v4.0.5 and mev-boost v1.5.0 Since Shapella hardfork I’ve missed all block proposals. We tried to find out the reason with Geth team (https://github.com/ethereum/go-ethereum/issues/27138), with Nishant from Prysm team and finally, I hope, find out. It seems something’s wrong with mev-boost. Here are logs. GETH

INFO [05-25|17:37:00.338] Starting work on payload                 id=0xbf229667f35a711a
INFO [05-25|17:37:00.354] Updated payload                          id=0xbf229667f35a711a number=17,336,201 hash=cc55fa..d601d5 txs=38   gas=3,174,724 fees=0.004864138539 root=8967e9..855f10 elapsed=16.021ms
INFO [05-25|17:37:02.385] Updated payload                          id=0xbf229667f35a711a number=17,336,201 hash=b17d61..8a1df9 txs=78   gas=6,407,568 fees=0.00885417138  root=09c947..4f22cd elapsed=27.736ms
INFO [05-25|17:37:04.412] Updated payload                          id=0xbf229667f35a711a number=17,336,201 hash=a515c2..da0139 txs=110  gas=8,137,497 fees=0.01486888797  root=e1f251..3b35bf elapsed=24.003ms
INFO [05-25|17:37:06.465] Updated payload                          id=0xbf229667f35a711a number=17,336,201 hash=246119..3bbc52 txs=124  gas=9,005,556 fees=0.01564731379  root=39784e..5459b9 elapsed=51.373ms
INFO [05-25|17:37:08.554] Updated payload                          id=0xbf229667f35a711a number=17,336,201 hash=60ab4e..3c5cb3 txs=155  gas=10,991,513 fees=0.01952288677  root=bd5e6a..67053f elapsed=88.920ms
INFO [05-25|17:37:10.623] Updated payload                          id=0xbf229667f35a711a number=17,336,201 hash=16fd55..86ea0d txs=181  gas=13,330,854 fees=0.02821441559  root=8102ff..d52e51 elapsed=64.576ms
INFO [05-25|17:37:11.563] Stopping work on payload                 id=0xbf229667f35a711a reason=delivery

PRYSM beacon-chain

[2023-05-25 17:37:11]  INFO rpc/validator: Received header with bid blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a builderPubKey=0xb3ee7afcf27f1f1259ac1787876318c6584ee353097a50ed84f51a1f21a323b3736f271a895c7ce918c038e4265918be value=74477060703171560
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix
[2023-05-25 17:37:18]  WARN removing invalid BLSToExecutionChange from pool error=withdrawal credential prefix is not a BLS prefix

PRYSM validator [2023-05-25 17:37:20] ERROR validator: Failed to propose block blockSlot=6516184 error=rpc error: code = Unknown desc = could not submit blinded block: error posting the SignedBlindedBeaconBlockCapella to the builder api: unsupported error code: 502: did not receive 200 response from API pubKey=0xa4bc0c220ccc

MEV-BOOST

time="2023-05-25T17:37:11+05:00" level=info msg="best bid" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a blockNumber=17336201 method=getHeader parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 pubkey=0xa4bc0c220ccc23a3a3b5dd6a6125396067405c3eadb0dd4a1b7de120cb2eceac5d9126bab0e6b4d78a952d8f86ee01d7 relays="https://0xb3ee7afcf27f1f1259ac1787876318c6584ee353097a50ed84f51a1f21a323b3736f271a895c7ce918c038e4265918be@relay.edennetwork.io" slot=6516184 txRoot=0xb160d636395a0a7e752fbb71df939a215fa75eaf4b623711ca5aaa67ed3d6ca7 value=0.074477060703171560 version=v1.5.0
time="2023-05-25T17:37:11+05:00" level=info msg="http: GET /eth/v1/builder/header/6516184/0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69/0xa4bc0c220ccc23a3a3b5dd6a6125396067405c3eadb0dd4a1b7de120cb2eceac5d9126bab0e6b4d78a952d8f86ee01d7 200" duration=0.219741 method=GET path=/eth/v1/builder/header/6516184/0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69/0xa4bc0c220ccc23a3a3b5dd6a6125396067405c3eadb0dd4a1b7de120cb2eceac5d9126bab0e6b4d78a952d8f86ee01d7 status=200 version=v1.5.0
time="2023-05-25T17:37:19+05:00" level=warning msg="error making request to relay, retrying" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 8429 ms into slot\"}\n" method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 slot=6516184 url="https://relay.edennetwork.io/eth/v1/builder/blinded_blocks" version=v1.5.0
time="2023-05-25T17:37:19+05:00" level=warning msg="error making request to relay, retrying" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 8759 ms into slot\"}\n" method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 slot=6516184 url="https://relay.edennetwork.io/eth/v1/builder/blinded_blocks" version=v1.5.0
time="2023-05-25T17:37:20+05:00" level=warning msg="error making request to relay, retrying" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 9114 ms into slot\"}\n" method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 slot=6516184 url="https://relay.edennetwork.io/eth/v1/builder/blinded_blocks" version=v1.5.0
time="2023-05-25T17:37:20+05:00" level=warning msg="error making request to relay, retrying" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 9458 ms into slot\"}\n" method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 slot=6516184 url="https://relay.edennetwork.io/eth/v1/builder/blinded_blocks" version=v1.5.0
time="2023-05-25T17:37:20+05:00" level=warning msg="error making request to relay, retrying" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 9803 ms into slot\"}\n" method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 slot=6516184 url="https://relay.edennetwork.io/eth/v1/builder/blinded_blocks" version=v1.5.0
time="2023-05-25T17:37:20+05:00" level=error msg="error making request to relay" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a error="max retries exceeded" method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 slot=6516184 url="https://relay.edennetwork.io/eth/v1/builder/blinded_blocks" version=v1.5.0
time="2023-05-25T17:37:20+05:00" level=error msg="no payload received from relay!" blockHash=0xbefbbc79d55b8b6fc2a4799ed9060617ae63f00555528183b9526625e588d68a method=getPayload parentHash=0xfd55df21f460c21b836b0ea0fbb03fb1cef06fa65f9d7f49d673df757c1c5d69 relays="https://0xb3ee7afcf27f1f1259ac1787876318c6584ee353097a50ed84f51a1f21a323b3736f271a895c7ce918c038e4265918be@relay.edennetwork.io" slot=6516184 version=v1.5.0