lnd: [bug]: "LNWL: unable to fetch transaction" error after upgrading to 0.16.1-beta

Background

Upgraded (built myself) to 0.16.1-beta, started to see unable to fetch transaction errors just after restart:

2023-04-26 20:58:49.259 [INF] LTND: Version: 0.16.1-beta commit=v0.16.1-beta, build=production, logging=default, debuglevel=info,LNWL=debug,HSWC=trace
2023-04-26 20:58:49.260 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2023-04-26 20:58:49.264 [INF] LTND: Pprof listening on 127.0.0.1:9000
2023-04-26 20:58:49.346 [INF] RPCS: RPC server listening on 127.0.0.1:10009
2023-04-26 20:58:49.370 [INF] RPCS: gRPC proxy started at 127.0.0.1:8080
2023-04-26 20:58:49.371 [INF] LTND: Opening the main database, this might take a few minutes...
2023-04-26 20:58:49.371 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=true
2023-04-26 20:58:49.379 [INF] CHDB: Compacting database file at /home/lnd/.lnd/data/graph/mainnet/channel.db
2023-04-26 20:59:53.309 [INF] CHDB: DB compaction of /home/lnd/.lnd/data/graph/mainnet/channel.db successful, 286453760 -> 193302528 bytes (gain=1.48x)
2023-04-26 20:59:53.310 [INF] CHDB: Swapping old DB file from /home/lnd/.lnd/data/graph/mainnet/temp-dont-use.db to /home/lnd/.lnd/data/graph/mainnet/channel.db
2023-04-26 20:59:53.460 [INF] CHDB: Compacting database file at /home/lnd/.lnd/data/chain/bitcoin/mainnet/macaroons.db
2023-04-26 20:59:53.486 [INF] CHDB: DB compaction of /home/lnd/.lnd/data/chain/bitcoin/mainnet/macaroons.db successful, 32768 -> 32768 bytes (gain=1.00x)
2023-04-26 20:59:53.491 [INF] CHDB: Swapping old DB file from /home/lnd/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.db to /home/lnd/.lnd/data/chain/bitcoin/mainnet/macaroons.db
2023-04-26 20:59:53.498 [INF] CHDB: Compacting database file at /home/lnd/.lnd/data/graph/mainnet/sphinxreplay.db
2023-04-26 21:00:13.311 [INF] CHDB: DB compaction of /home/lnd/.lnd/data/graph/mainnet/sphinxreplay.db successful, 47603712 -> 42123264 bytes (gain=1.13x)
2023-04-26 21:00:13.319 [INF] CHDB: Swapping old DB file from /home/lnd/.lnd/data/graph/mainnet/temp-dont-use.db to /home/lnd/.lnd/data/graph/mainnet/sphinxreplay.db
2023-04-26 21:00:13.391 [INF] LTND: Creating local graph and channel state DB instances
2023-04-26 21:00:13.429 [INF] CHDB: Checking for schema update: latest_version=29, db_version=29
2023-04-26 21:00:13.431 [INF] CHDB: Checking for optional update: prune_revocation_log=false, db_version=0: prune revocation log
2023-04-26 21:00:13.432 [INF] LTND: Database(s) now open (time_to_open=1m24.053012836s)!
2023-04-26 21:00:13.432 [INF] LTND: Attempting automatic wallet unlock with password provided in file
2023-04-26 21:00:17.453 [INF] LNWL: Opened wallet
2023-04-26 21:00:17.826 [INF] CHRE: Primary chain is set to: bitcoin
2023-04-26 21:00:37.985 [ERR] LNWL: unable to fetch transaction <redacted> for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
2023-04-26 21:00:51.137 [ERR] LNWL: unable to fetch transaction <redacted> for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
2023-04-26 21:00:51.157 [ERR] LNWL: unable to fetch transaction <redacted> for mempool: -5: No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
<more of those [ERR]>...

Restarted lnd agian, restarted bitcoind - did not help.

I do see this line in bitcoind log thought:

BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications

Maybe this could be related? Anyway, previous lnd version worked.

Your environment

• 0.16.1-beta from git
• Linux odroid-hc1 6.1.0-0.deb11.6-armmp-lpae #1 SMP Debian 6.1.15-1~bpo11+1 (2023-03-16) armv7l GNU/Linux
• bitcoind v24.0.1
• built with Go 1.19.7

Steps to reproduce

Not sure, I just built and restarted lnd.

Expected behaviour

lnd shoud start successfully without spamming logs with [ERR].

Actual behaviour

lnd spams log with unable to fetch transaction and seems to stop after that.