snarkOS: Aleo node stuck block 882720 issues

🐛 Bug Report

It seems I am trying to understand that when the next block loses the proof parameters, our node cannot synchronize the block and is always stuck at the current block 882720

snarkos commit hash

commit 4896a1200a4605d1de6fe6cb53e1efa9ccdb6152

log

Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.506498Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '146.59.118.20:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.525679Z  INFO snarkos_node_bft::sync: Syncing the BFT to block 882721...
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.526355Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.526407Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '2730624083708228..' for round 1804738 with 3 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.539051Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '2730624083708228..' from block 882721 - Failed to provide a transmission for round 1804738 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.539442Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.539524Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.539540Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7793278040859612..' for round 1804738 with 5 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.552663Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7793278040859612..' from block 882721 - Failed to provide a transmission for round 1804738 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.552996Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.553063Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.553078Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1503313062419249..' for round 1804738 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.565870Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1503313062419249..' from block 882721 - Failed to provide a transmission for round 1804738 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.566178Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.566241Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.566264Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5268222357895292..' for round 1804738 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.578946Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5268222357895292..' from block 882721 - Failed to provide a transmission for round 1804738 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.579230Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.579289Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.579306Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7957666768221182..' for round 1804739 with 5 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.592017Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7957666768221182..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.592315Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.592373Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.592389Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '3189623774571386..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.605062Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '3189623774571386..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.605351Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.605429Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.605446Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7767893619424610..' for round 1804739 with 5 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.614994Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '202.94.169.43:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.615050Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '202.94.169.43:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.617034Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '212.126.35.133:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.617064Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '212.126.35.133:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.618146Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7767893619424610..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.618376Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.618442Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.618458Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5321463306864422..' for round 1804739 with 5 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.631076Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5321463306864422..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.631318Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.631368Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.631385Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5206384593268982..' for round 1804739 with 5 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.643023Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '34.85.217.171:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.643069Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '34.85.217.171:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.644054Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5206384593268982..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.644330Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.644376Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.644390Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5461985580841839..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.657110Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5461985580841839..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.657392Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.657454Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.657478Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '4904938533001294..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.659132Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '162.219.87.220:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.659178Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '162.219.87.220:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.670192Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '4904938533001294..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.670544Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.670616Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.670662Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1693397399859500..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.673638Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '162.55.4.41:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.673685Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '162.55.4.41:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.683591Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1693397399859500..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.683970Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.684038Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.684061Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '3529370082468350..' for round 1804739 with 3 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.696983Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '3529370082468350..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.697425Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.697466Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '2481462369164066..' for round 1804739 with 2 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.710142Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '2481462369164066..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.710435Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.710484Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.710499Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5438623388537520..' for round 1804739 with 3 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.723184Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5438623388537520..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.723547Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.723596Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.723609Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '6201130001305036..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.736227Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '6201130001305036..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.736630Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.736691Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.736713Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7246563275654527..' for round 1804739 with 3 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.749268Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7246563275654527..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.749677Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.749726Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.749780Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '2099736442332576..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.762519Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '2099736442332576..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.762763Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.762812Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.762844Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1440169275662735..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.776298Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1440169275662735..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.776642Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.776689Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.776717Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '6750025850424811..' for round 1804739 with 5 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.789241Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '6750025850424811..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.789608Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.789671Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.789707Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '4623107476002895..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.802564Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '4623107476002895..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.803056Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle12vvgrq4v58zufy88nrp5pqs0mmfn3nhyk72jg06dhqzxm9g6y3y3pv79hlu5x5eyerawze922m4gqd7d2ev in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.803125Z ERROR snarkos_node_bft::helpers::storage: Missing solution puzzle1zjyt9etff9y8td5vlpjnef87v25zt0094urxnfv83amvf82rcrx9fj33hw85zag4yshl8qcvk4mqz6842l0 in block 882721
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.803162Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5884107348578819..' for round 1804739 with 4 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.816498Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5884107348578819..' from block 882721 - Failed to provide a transmission for round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.816794Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7161353632625381..' for round 1804739 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.843506Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7161353632625381..' from block 882721 - Missing previous certificate '2730624083708228..' for certificate in round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.843962Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '3158757284400046..' for round 1804739 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.844183Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'WorkerPing' from '37.187.71.86:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.856617Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '15.188.23.173:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.856660Z TRACE snarkos_node_bft::primary: Skipping a certified batch from '15.188.23.173:5000' (node is syncing)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.864832Z TRACE snarkos_node_consensus: Received unconfirmed transaction 'at13lcsdhtzade50..' in the queue
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.870050Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '3158757284400046..' from block 882721 - Missing previous certificate '2730624083708228..' for certificate in round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.870681Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '3589188532721531..' for round 1804739 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.897138Z TRACE snarkos_node_consensus: Received unconfirmed solution 'puzzle1kmpwjqwd4..' in the queue
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.901243Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '3589188532721531..' from block 882721 - Missing previous certificate '2730624083708228..' for certificate in round 1804739 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.901573Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1688346477348163..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922033Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1688346477348163..' from block 882721 - Missing certificates for the previous round 1804739 in storage (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922259Z  INFO snarkos_node_bft::sync: Syncing the BFT to block 882722...
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922625Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922662Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '8434304740907216..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922730Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '8434304740907216..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922934Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922955Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '2511646112911735..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.922992Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '2511646112911735..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923119Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923136Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '2256278529617930..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923169Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '2256278529617930..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923350Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923367Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '5007596053265549..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923400Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '5007596053265549..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923597Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923617Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '8073076794180332..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923651Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '8073076794180332..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923848Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923879Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1657083653759284..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.923922Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1657083653759284..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924435Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924455Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1877864360451902..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924488Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1877864360451902..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924586Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924598Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1629618136340475..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924624Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1629618136340475..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924880Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924894Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '8297501672594947..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.924918Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '8297501672594947..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925063Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1fxujr87k6sv8t49dh3qea2d2d9qy3a87ftg6pug85ayudrh5jsys3nrakd in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925078Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7097770532189334..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925101Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7097770532189334..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925319Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1zj0w4d29fcr42vhkva2j00m7jquy8cjfvm96xhl57q7sdpa8tgqq26wgkp in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925332Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7919037659748118..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925358Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7919037659748118..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925970Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1zj0w4d29fcr42vhkva2j00m7jquy8cjfvm96xhl57q7sdpa8tgqq26wgkp in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.925986Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1352588827536614..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926011Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1352588827536614..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926102Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1zj0w4d29fcr42vhkva2j00m7jquy8cjfvm96xhl57q7sdpa8tgqq26wgkp in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926110Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '6406351278446954..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926134Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '6406351278446954..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926245Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1zj0w4d29fcr42vhkva2j00m7jquy8cjfvm96xhl57q7sdpa8tgqq26wgkp in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926254Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '1753086907267519..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926278Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '1753086907267519..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926365Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1zj0w4d29fcr42vhkva2j00m7jquy8cjfvm96xhl57q7sdpa8tgqq26wgkp in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926377Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '7503258450864172..' for round 1804740 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926402Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '7503258450864172..' from block 882722 - Failed to provide a transmission for round 1804740 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926595Z  WARN snarkos_node_bft::helpers::storage: Missing transaction at1zj0w4d29fcr42vhkva2j00m7jquy8cjfvm96xhl57q7sdpa8tgqq26wgkp in block 882722
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926621Z DEBUG snarkos_node_bft::helpers::storage: Syncing certificate '3156702872472583..' for round 1804741 with 1 transmissions
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.926655Z ERROR snarkos_node_bft::helpers::storage: Failed to insert certificate '3156702872472583..' from block 882722 - Failed to provide a transmission for round 1804741 (gc = 1804690)
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.927336Z  INFO snarkos_node_bft::bft: Proceeding to commit round 1804740 with leader aleo14yr9gw824yp95fpwfvyq0q5zka088g530chltrsvpsjpusqupgrqlgcu84...
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.928096Z  WARN snarkos_node_bft::gateway: [MemoryPool] Disconnecting from '162.19.62.176:5000' - Sync - BFT failed to retrieve transmission 'puzzle12vvgrq4v5..' from round 1804738
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.940540Z TRACE snarkos_node_bft::gateway: [MemoryPool] Received 'BatchCertified' from '162.19.62.176:5000'
Dec 10 20:40:03 10-7-167-1 snarkos[4002800]: 2023-12-10T12:40:03.940598Z TRACE snarkos_node_bft::gateway: [MemoryPool] Sending 'Disconnect' to '162.19.62.176:5000'

Environment

snarkOS Version: v2.2.5 Rust Version: rustc 1.74.0 (79e9716c9 2023-11-13) OS: Ubuntu 22.04.3 LTS