lighthouse: Investigate PeerDAS KZG library stack overflow during block production
Description
We’ve recently switched from the c-kzg library to the Rust peerdas-kzg library on the das branch (#5941). The purpose for the switch was to:
- Try out a new alternative to c-kzg;
- Allow us to merge
dasbranch tounstablewithout having to updatec-kzgto a development version
However we’ve recently ran into some issues during interop testing after the change, and we’re getting stack overflow panics, likely during block production:
Jul 15 10:03:55.007 DEBG Fork choice successfully updated before block production, fork_choice_slot: 60, slot: 60, service: beacon
Jul 15 10:03:55.007 DEBG Not attempting re-org reason: head not late, service: beacon
Jul 15 10:03:55.008 ERRO No valid eth1_data votes, `votes_to_consider` empty, outcome: casting `state.eth1_data` as eth1 vote, genesis_time: 1721037115, earliest_block_timestamp: 1721037095, lowest_block_number: 0, service: deposit_contract_rpc
Jul 15 10:03:55.008 DEBG Produced vote for eth1 chain block_hash: 0xcc93bda48e1227e94e768ee757160974267c72c872f4c89213fbee552bb2ecf2, deposit_count: 0, deposit_root: 0xd70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e, service: deposit_contract_rpc
Jul 15 10:03:55.008 DEBG Issuing engine_getPayload parent_hash: 0xd32b8265b9e5694d48a42d14175d2692fdb918d305b91d336f2fd2c3167a8df5, timestamp: 1721037835, prev_randao: 0x4c74c85276c06a77410fb641d54224b09045bd03cf19c131ac158b534cd31ad6, suggested_fee_recipient: 0x8943545177806ed17b9f23f0a21ee5948ecaa776, service: exec
Jul 15 10:03:55.011 DEBG Processed HTTP API request method: GET, path: /eth/v1/node/version, status: 200 OK, elapsed: 86.916µs
Jul 15 10:03:55.103 DEBG Produced block on state block_size: 2141, service: beacon
Jul 15 10:03:55.115 DEBG Processed HTTP API request method: GET, path: /eth/v2/validator/blocks/60, status: 200 OK, elapsed: 108.519167ms
Jul 15 10:03:55.115 DEBG Processed HTTP API request method: GET, path: /eth/v1/config/spec, status: 200 OK, elapsed: 104.2355ms
Jul 15 10:03:55.117 DEBG Processed HTTP API request method: GET, path: /eth/v1/node/syncing, status: 200 OK, elapsed: 199.834µs
Jul 15 10:03:55.257 DEBG Signed block received in HTTP API slot: 60
thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
Kurtosis config to reproduce:
participants:
# Supernodes
- cl_type: prysm
cl_image: ethpandaops/prysm-beacon-chain:peerdas-initial-sync
cl_extra_params: [--subscribe-all-subnets]
- cl_type: lighthouse
cl_image: ethpandaops/lighthouse:das-devnet-2
cl_extra_params: [--subscribe-all-data-column-subnets]
# Non supernodes
- cl_type: prysm
cl_image: ethpandaops/prysm-beacon-chain:peerdas-initial-sync
- cl_type: lighthouse
cl_image: ethpandaops/lighthouse:das-devnet-2
network_params:
eip7594_fork_epoch: 0
eip7594_fork_version: "0x50000038"
snooper_enabled: true
global_log_level: debug
additional_services:
- dora
- goomy_blob
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 33 (33 by maintainers)
Added a test here showing the CI failing even if we use vectors as return types, and then removing the stack allocation being done inside of map and showing it passing.
The stack allocation is akin to the
let blob = KzgBlob::from_bytes(blob).map_err(KzgError::from)?;line in the original codebase.My understanding (could be wrong) was that in the following code:
into_par_iterwas using the global rayon pool, while the call to compute_cells_and_proofs will invoke the peerdas-kzg lib and use the local rayon threadpool that peerdas-kzg instantiated, which would give rise to that issue – does that sound congruent with your thoughts?yep your image actually revealed a bug we’ve had for a while! We rarely run validator client in debug mode so this bug has been around for a while without us noticing! Fixed in #6125
For the latest commit (04d9eef8a) the logs for vc-geth-lighthouse was :
My current hypothesis is that the lengthy periods of time are due to retries. This is because in the error messages by Kurtosis where the service did not start up, it notes the retry attempts:
If this is correct, then the long startup times on
55a3be72band04d9eef8awould be due to Kurtosis retrying with the rust-embed issue, and the long startup times on018f38257would possibly be due to the Kurtosis retrying on theUnable to initialize validator configissueLooks like a panic inside
publish_block(after the block is produced)