@Roasbeef not sure but I think this problem will affect way more people having to rescan their wallets if they figure out that their anchor was swept. Because I was sweeping a lot of Anchor Outputs and effectively also RBF existing Anchor-Sweeps which means way more people have a pending tx in their lnd wallet which will never be confimed.

@Roasbeef - it also looks like LND does not check if a channel funding transaction is valid (neither a local one, nor a remote channel funding transaction).

I was wondering how it’s possible to have pending opens for weeks (iniitiated by the remote party) where the funding transaction can’t be found in the mempool. It seems that those peers tried to open a channel with an invalid funding transaction and my node not even realizing that the channel opening was basically a double-spend that should have been rejected right away.

Just an update for anyone affected by this. To get everything in order again you must reset your wallet balance so that LND completely rescans the blockchain for transactions.

1. Restart LND and reindex wallet transactions

Insert this line in [Application Options]

reset-wallet-transactions=true

and restart LND. Make sure to remove this line afterwards, otherwise LND will do this every time on startup.

It will take a while, depending on your hardware and bitcoind setup. Only do this if you’re not running a pruned bitcoind - then you’re in uncharted territory (i.e. I don’t know if it will work - it probably won’t).

2. Abandon the affected channels

If you do have pending channels that used the already spent anchor outputs (where the funding transaction was rejected by the network) you can abandon those channels to make them disappear from the pending_channels list. Make sure that these are really “phantom” channels, don’t abandon any channels that are just waiting for confirmations.

Hi y’all, I’ve confirmed that this PR fixes the issue!

I was able to properly reproduce the scenario deterministically in an integration test, to confirm my commit fixed the core issue. So thanks to y’all for such a detailed issue report! I think the fix I added to the sweeper may actually resolve some other issues in the tracker related to unconfirmed spends.

As 0.15 is still about a month away, I think maybe it makes sense to package this in an 0.14.3 minor release to alleviate this strain. Thoughts?

but yeah if this functionality is ready why not ship with 0.14.3

Def, we may look to package up some other bug fixes as well while we’re at it. Earliest possible rc would likely be mid/late next week.

I would agree. The earlier the better. Thank you for the quick fix!

I was wondering how it’s possible to have pending opens for weeks (iniitiated by the remote party) where the funding transaction can’t be found in the mempool.

If you extend lnd a channel (you’re on the receiving side), then it can’t actually verify the funding transaction, as it only sees the outpoint. We don’t watch the mempool, only the chain. After the channel has been confirmed, we’re able to fully validate the funding transaction. See this issue for adding more pre-broadcast mempool for funding transactions.

We wait for 14 days before automatically removing zombie channels: https://github.com/lightningnetwork/lnd/blob/28ea2736a04566b06c38533e5e48e1b8d185a49b/funding/manager.go#L101. We also restrict the number of pending channels a given peer can open to us as well.

In practice I don’t think this should require doing a full rescan. With the way the sweeper works, when it tries to publish something invalid, it’ll just periodically back off and retry again, possibly with a distinct output set. We also watch for when that output is spent and then stop sweeping and signal back to the caller. So if no unconfirmed dependent spends exist, then it should just go away.

When you do a funding attempt, all the inputs are locked, which rn doesn’t show up in the normal balance call. If we detect an invalid spend, then the transaction is removed from the wallet, and eventually the locked funds will become available again. What we don’t do rn is automatically remove that transaction from the pending channels list, which is fixed in this PR.

So I think the missing gap here is:

• When the sweeper encounters an output that was spent, and not by us, then it should report back to the wallet to remove that transaction.

With the way the wallet interacts rn with RBF transaction, it doesn’t know which one will actually confirm (the mempool is a murky place), so it’s possible a user attempts to spend one of those outputs as zero conf. When/if a valid transaction eventually confirms the wallet state will be cleaned up, but only if it’s actually one of our spends (need to verify this).

I didn’t check, but I think this happened at a time with high on-chain fees, and the close tx was already confirmed (so grabbing the anchor doesn’t help with that).

We’ve seen a script attempt to more aggressively sweep anchor outputs, they’d claimed $70 or so worth when I last checked. Right now we always use the smallest fee rate possible when attempting to sweep anchor outputs. You’ll be able to set the anchor fee rate after this PR: https://github.com/lightningnetwork/lnd/pull/6025

Re that balance snapshot, when UTXOs are locked, they don’t show up in that base balance output. We should likely add another field there though, so it’s a bit easier to keep track of things.

The open TX is not broadcast, this is in my logs: [ERR] FNDG: Unable to broadcast funding tx xxx for ChannelPoint(xxx:1): transaction rejected: output already spen

This PR fixes that behavior: https://github.com/lightningnetwork/lnd/pull/5158

So I think the missing gap here is: When the sweeper encounters an output that was spent, and not by us, then it should report back to the wallet to remove that transaction.

I’ve implemented this here: https://github.com/lightningnetwork/lnd/pull/6274. Needs tests, but the approach should work, I think there’re other areas we can apply the same approach to make accounting a bit easier, and also cut down on unnecessary transactions piling up in ListTransactions.

Do we have a commitement tx in our local memory when a channel is opened to us although we still have no funds on our side

When we’re the receiver of a new funding attempt? Yes, we’ll write that to disk as confirmation may happen after we’re down.

And if yes how do we forget about the channel by broadcasting the commitment tx or are we just purging it in our database?

When we’re the receiver of a new channel, and we choose to drop the channel, we’ll just remove the channel state (it goes into a closed channel log). Broadcasting the commitment won’t do anything if the funding transaction actually doesn’t exist, or can never confirm for some reason.

We only do this is we’re not the initiator though.

Related to this Topic, should we make the Anchor Sweep Transaktion (non-RBF Signaling) in that case nobody could RBF the funds even after they are broadcasted or does this imply even more logic for handling funds in case there are no other outputs left in the wallet. Might increase the complexity of the wallet so I understand why we RBF the Anchor Sweep in the first place but just bringing it up here as an idea.