lnd: Possibly missing updates from Bitcoind backend

Background

I have 9 lnd’s sharing a single Bitcoind backend on a single server. I didn’t open a channel with any of them in order to see this issue. I funded the first address generated from each lnd’s wallet with some satoshis(which confirmed), and destroyed each lnd. Then, I restored each using the seed phrase, with multiple re-synchronizing from Bitcoind at once. Some of them (it seems the first ones to get a connection to Bitcoind) restored fine, with funds in their wallet. Others (the later ones to connect) seemed to miss their funding transactions. The address look-ahead window should have caught the funding transactions, as the addresses I sent to were the first generated addresses for each seed.

Your environment

• version of lnd = 0.4.1-beta commit=09c997983127bb1e575d97c69aa364a82cfb1df7
• which operating system (uname -a on *Nix) = Linux provingapoint 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• version of btcd, bitcoind, or other backend = Bitcoin Core Daemon version v0.16.0.0-g4b4d7eb255
• any other relevant environment details = All running in Docker containers, which probably exacerbated the connectivity issues.

I’ve confirmed the wallet starts recovery without issue:

2018-05-03 05:18:07.589 [WRN] LTND: open /root/.lnd/lnd.conf: no such file or directory
2018-05-03 05:18:07.589 [INF] LTND: Version 0.4.1-beta commit=09c997983127bb1e575d97c69aa364a82cfb1df7
2018-05-03 05:18:07.589 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2018-05-03 05:18:07.592 [INF] CHDB: Checking for schema update: latest_version=0, db_version=0
2018-05-03 05:18:07.593 [INF] RPCS: Generating TLS certificates...
2018-05-03 05:18:07.616 [INF] RPCS: Done generating TLS certificates
2018-05-03 05:18:07.623 [INF] RPCS: password RPC server listening on 127.0.0.1:10009
2018-05-03 05:18:07.624 [INF] RPCS: password gRPC proxy started at 127.0.0.1:8080
2018-05-03 05:18:07.624 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create wallet, or `lncli unlock` to unlock already created wallet.
2018-05-03 05:18:50.518 [INF] LNWL: Opened wallet
2018-05-03 05:18:50.519 [INF] LTND: Wallet recovery mode enabled with address lookahead of 250 addresses
2018-05-03 05:18:50.614 [INF] LTND: Primary chain is set to: bitcoin
2018-05-03 05:18:50.614 [INF] LTND: Initializing bitcoind backed fee estimator
2018-05-03 05:18:51.939 [INF] LNWL: Opened wallet
2018-05-03 05:18:51.944 [INF] LNWL: Started listening for blocks via ZMQ on tcp://bitcoind:29000
2018-05-03 05:18:53.281 [INF] LNWL: The wallet has been unlocked without a time limit
2018-05-03 05:18:53.283 [INF] LNWL: Catching up block hashes to height 520983, this will take a while...
2018-05-03 05:18:53.285 [INF] LTND: LightningWallet opened
2018-05-03 05:18:53.286 [INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=250
2018-05-03 05:18:53.292 [INF] LNWL: Caught up to height 0

Steps to reproduce

Connect multiple lnds to a single Bitcoind backend+ZMQ, and restore them from some seeds with confirmed funds to an address derived from the seed.

Alternatively, I think if you drop some crucial packets on ZMQ you’ll see similar results.

Expected behaviour

LND is aware when it’s missing crucial information from its backend, and it recovers from data connectivity issues gracefully.

Actual behaviour

Connectivity issues with the Bitcoind backend appear to “trick” LND into the wrong state with regards to funding and possibly seeing crucial transactions.