lnd: Falling out of sync on testnet because of "possible reorg"

Background

I’m running lnd on testnet, and I noticed that it falls out of sync each time a new block is mined as it sees a “possible reorg”

Your environment

version of lnd lnd version 0.6.0-beta commit=v0.6.0-beta-118g1acd38e48c168b86c291524eb56b8fcdf04a910c
which operating system (uname -a on *Nix) Linux Celeri 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
version of btcd, bitcoind, or other backend Bitcoin Core Daemon version v0.17.1
any other relevant environment details Here is one example of what I get in the logs:

2019-05-09 01:37:00.174 [DBG] LNWL: Possible reorg at block 00000000000000000021a9f98b1b5d5d9d3017a780a74561173d26f6d47d81ca
2019-05-09 01:37:00.177 [ERR] LNWL: Unable to process chain reorg: -5: Block not found
2019-05-09 01:37:00.180 [DBG] LNWL: Possible reorg at block 00000000000000000021a9f98b1b5d5d9d3017a780a74561173d26f6d47d81ca
2019-05-09 01:37:00.181 [DBG] LNWL: Possible reorg at block 00000000000000000021a9f98b1b5d5d9d3017a780a74561173d26f6d47d81ca
2019-05-09 01:37:00.182 [ERR] LNWL: Unable to process chain reorg: -5: Block not found
2019-05-09 01:37:00.183 [ERR] LNWL: Unable to process chain reorg: -5: Block not found

Interestingly the blockhash indeed doesn’t exist

Steps to reproduce

Just run lnd, it will first catch up normally and then fall out of sync when the next block is mined.

Expected behaviour

Lnd should stay synced

Actual behaviour

I need to turn it off and start it again to keep it synced.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

Hi I have not been running lnd for a while now, but indeed what you say makes sense since I mentionned running both mainnet and testnet. It would also explained why I didn’t have the issue on my BTCPay since I was running it on another server.

I nuked the server I was using at the time so there’s no way I can confirm this but it is very likely I messed up the zmq port configuration.

Thanks!

Sosthene00 on Apr 3, 2021

I encountered this same issue and can confirm the cause to be running mainnet and testnet bitcoind backends while also not updating the bitcoind.zmqpubrawblock and bitcoind.zmqpubrawtx to be on different ports between mainnet and testnet. This will cause lnd to get updates from both backends, causing the issue of going out of sync.

You should be able to confirm this issue with the OP as well. They mention running testnet but if you look at the logs, the block lnd is complaing about is a valid mainnet block. 00000000000000000021a9f98b1b5d5d9d3017a780a74561173d26f6d47d81ca

2019-05-09 01:37:00.180 [DBG] LNWL: Possible reorg at block 00000000000000000021a9f98b1b5d5d9d3017a780a74561173d26f6d47d81ca

Which is mainnet block 575190 from 2019-05-08 19:36 and occurring very near to OP.

I spent a little time investigating this issue myself and hope this helps someone else save a little time in the future!

cryptosharks131 on Apr 1, 2021