lnd: [bug]: crash-loop on startup: unmatched backend error: -26: mempool min fee not met

just a little more context on my situation for others to learn from:

• My lnd had sent some 1 sat/byte transactions that were unconfirmed
• I was running a default bitcoind mempool (300mb)
• Mempool spiked and started purging low-fee transactions, including mine
• I do an LND restart but get into a crash loop with mempool min fee not met
• I stop bitcoind, set maxmempool=1000 in bitcoind.conf, which increased my mempool size to 1000mb. restart bitcoind
• Now LND is able to restart fine
• However, those original 1 sat/byte transactions were dropped from my LND wallet, so their input utxos were now available to spend, and I spent them again elsewhere

In general my takeaway is that a lot of problems can be avoided if you just run a larger bitcoind mempool. but there are caveats associated with this

I don’t think we should do that either. It will just encourage users to actually do RBF. There were at least two cases I know of where users imported lnd keys into Electrum to create an RBF replacement TX for a funding transaction, putting their funds at risk. This only worked because the original one was already evected from the mempool. But IMO we shouldn’t encourage being able to do that while the original TX is still present. Also the “RBF enabled” label on a block explorer might give users wrong ideas too.

Double spending the broadcasted, but unconfirmed and possibly out of (most) mempools, transactions via lnd would be cool. To do something like this now, I figure you’d need to create and broadcast a transaction using the same utxo manually outside of lnd? And then once lnd detects the double spend, it would remove the pending channel and opening transaction?

Correct. We recently added this to chantools, but it would be cool to have that option within lnd as well.

Duplicate issue from the last time we had a fee spike: https://github.com/lightningnetwork/lnd/issues/5969/. Other instances across time: https://github.com/lightningnetwork/lnd/issues?q=is%3Aissue+mempool+min+fee+not+met+is%3Aclosed.

I was able to unblock by going into my bitcoin.conf and increasing maxmempool, but I don’t think that is a sustainable solution.

There’s not a good way around it afaict: bitcoind will start to increase the min fee needed to get into the mempool and if you want to allow things lower than that to get in, you need to increase the max mempool size.

We likely should properly catch this error and proceed. However that would sort of mask the issue here, since ppl would wonder why their transactions aren’t getting mined.

I would do the following:

For case 1, if we do this, then I think we should have a clearer error message to force the user to action. So something like “consider raising your maxmempool value to xxx GB”. This way they know what to do and we can just close any other incoming issues, saying “works as expected”.

Maybe change the log-level for the min-relay-fee not met error, to not spam the log with errors and give the user the possibility via an RPC command to fetch the rawTx for this Crib-Transaction?

I’m on board with having the relevant RPCs also allow users to get the raw hex. We log it each time, but logs can be spammy at times on trace/debug, so it might be hard for users to find.

@ziggie1984

so it might be good to keep failing the startup in this case because then by increasing the mempool we make sure that we can CPFP the Commitment? Wdy

Yeah tough choice, see my message above. Either way things are sort of hosed…we can choose to force user intervention or handle it in the background and pray to satoshi that things drop enough to let the tx enter the mempool of nodes with default policy.

. We mark the Channel as StateCommitmentBroadcasted which is not quite right if we consider that the commitment was not broadcasted. So I think we should only mark the channel as StateCommitmentBroadcasted if the transaction successfully got placed in the mempool. Now problem is that we are not gonna rebroadcast the transaction as is, because we remove it and the User has to restart his node to trigger another retry

The issue here is that we never really know if the transaction is “in the mempool” or not. Node policy is less uniform these days, and a big transaction spike can quickly cause the min relay fee to jump up.

y. So I propose to keep the Channel in the state StateBroadcastCommit and maybe return the CommitmentTx in the lncli pendingchannels cmd to allow the user manually broadcast it again if he sees the mempool-minfee go down?

I think it’s a good idea to optionally allow RPC calls to print out the full hex of the broadcasted transaction so users can rebroacast on their own. Note that we’ll now also rebroadcast ourselves, so if the min relay fee drops down, then things should propagate.

I think for the second case we should not remove the transaction but keep trying rebroadcasting it (but change the log level for the min-mempool-fee failure, or give the user the possibility to fetch the rawtx in case he wants to broadcast it via other transaction relayers where the purging fee is lower?

The way the rebroadcaster works, it’ll keep going until things confirm or the sweeper removes it as a conflict is mined. So on start up, those should be broadcast again.

So to resolve this issue, we just need to make sure that on start up, we don’t refuse to start if we get this error.

See this issue: https://github.com/lightningnetwork/lnd/issues/4616

This code block needs to be updated to catch this error: https://github.com/btcsuite/btcwallet/blob/50978fcf79f8100eb62c185606c66367cc03b03e/wallet/wallet.go#L3463

With this recent PR, transactions should always be above the min relay fee at the time of tx creation. It’s possible that things drop below that value, but until wide spread package relay, there’s really nothing we can do about that.

The choice is essentially to:

• Silently eat this error to ensure we can start up, but the users transaction is hung and won’t confirm for some time.
• Block start up with the error, to force the user to increase their bitcoind settings to allow it to enter their mempool.

Note that even with the second option, the transaction won’t propagate since nodes will send a feefilter message telling peers to not send them transactions below that fee rate value.

Ok I analysed the situation regarding the startup issue and my analysis showed exactly 2 cases where we end up not starting because we could not broadcast the transaction (mempool-fee was not enough).

1. First situation is where we Force-Close a channel and the Force-Close Transaction is not broadcasted. https://github.com/lightningnetwork/lnd/blob/bbbf7d33fb1527acebb44e2a69d16fbcf24cc2fa/contractcourt/channel_arbitrator.go#L994.

So one downside of dropping this transaction is in an Anchor-Channel Case that we will not be able to FeeBump the transaction because the sweeper which sweeps the anchor (CPFP the Commitment) will fail with the missing input error so it might be good to keep failing the startup in this case because then by increasing the mempool we make sure that we can CPFP the Commitment? Wdyt? ⚠️

Another way would be:

The solution as @Roasbeef described would be to just include a case for the minrelay broadcast fail and proceed, but there is a small problem. We mark the Channel as StateCommitmentBroadcasted which is not quite right if we consider that the commitment was not broadcasted. So I think we should only mark the channel as StateCommitmentBroadcasted if the transaction successfully got placed in the mempool. Now problem is that we are not gonna rebroadcast the transaction as is, because we remove it and the User has to restart his node to trigger another retry. So I propose to keep the Channel in the state StateBroadcastCommit and maybe return the CommitmentTx in the lncli pendingchannels cmd to allow the user manually broadcast it again if he sees the mempool-minfee go down? Or maybe not remove it and keep rebroadcasting it, but then we should think of a way to hide the error because it would fill the logs and disturb the user ?

1. Second scenario is when we have the tweakless channel type (STATIC_REMOTE_KEY) and a timeout-htlc which we cannot broadcast because of the min-relayfee when restarting. Relevant codepart is here: https://github.com/lightningnetwork/lnd/blob/bbbf7d33fb1527acebb44e2a69d16fbcf24cc2fa/contractcourt/utxonursery.go#L291 which fails here: https://github.com/lightningnetwork/lnd/blob/bbbf7d33fb1527acebb44e2a69d16fbcf24cc2fa/contractcourt/utxonursery.go#L659

I think for the second case we should not remove the transaction but keep trying rebroadcasting it (but change the log level for the min-mempool-fee failure, or give the user the possibility to fetch the rawtx in case he wants to broadcast it via other transaction relayers where the purging fee is lower?

Happy to receive feedback in which direction we wanna go, if we have consensus I will code it.

I would do the following:

For case 1 where we are not able to broadcast the Commitment Transaction, still fail lnd to start to let the node-runner know that with the current mempool architecture his Commitment Transaction will not be able to confirm (because the anchor cannot bump a non-existent parent output (without package-relay)

For the second case I would lnd continue to startup successfully but rebroadcast the crib-transaction in a timely manner. Maybe change the log-level for the min-relay-fee not met error, to not spam the log with errors and give the user the possibility via an RPC command to fetch the rawTx for this Crib-Transaction?

Closing as the config issue has been resolved.

say my bitcoin node now runs an extra large mempool, but all its peers run default mempools. i could get into a situation where my bitcoin node has my txs, but nobody else does, even when mempool congestion chills out.

Correct, the mempool is truly a murky place unfortunately. Bitcoin nodes have something called a feefilter: as they start to increase their min fee amount, they’ll tell their peers not to send things below that fee rate.

The only solution to something like this in site is the fabled package relay construction. This would let you send that transaction, then also couple w/ it something to allow the fee to be high enough for acceptance.