lnd: [bug]: Consider failing HTLC backwards before upstream claims on-chain
Background
Consider an HTLC chain
Peer A -> Peer B -> Offline Peer
And assume Peer B Force Closes on Offline Peer due to HTLC missing in remote commitment upon expiry of HTLC.
The Force Close transaction is stuck in mempool for 144 blocks (CLTV delta of Peer B)
Now after 144 blocks, the peer A will also force close on peer B just because peer B has not failed the HTLC backward.
This causes a cascade of FC in current mempool (and specially with peers with shorter CLTV delta).
There is a similar case with LDK -> https://github.com/lightningdevkit/rust-lightning/issues/2275
Logs: Peer B force closes on an offline peer after HTLC expiry.
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-08 06:18:56.124 [INF] CNCT: ChannelArbitrator(0b228050fd8eeecf22073086a8885faf0c4d2bc02ad9480f23767368da411905:0): immediately failing htlc=3837313232636338363662613938653434353830343430613764383636646666 from remote commitment
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-08 06:18:56.204 [INF] CNCT: ChannelArbitrator(0b228050fd8eeecf22073086a8885faf0c4d2bc02ad9480f23767368da411905:0): force closing chan
The force close transaction is still in mempool. 144 blocks later peer A also force closed in a cascade
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.021 [INF] CNCT: Unilateral close of ChannelPoint(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0) detected
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.048 [WRN] HSWC: ChannelLink(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0): remote peer has closed on-chain
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.050 [INF] HSWC: ChannelLink(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0): exited
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.050 [INF] CNCT: ChannelArbitrator(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0): remote party has closed channel out on-chain
The second force close would have been prevented if HTLC was failed backward by peer B after force close with Offline Peer.
Your environment
-
version of
lnd“version”: “0.16.2-beta commit=v0.16.2-beta”, -
which operating system (
uname -aon *Nix)
Linux umbrel 5.10.17-v8+ #1421 SMP PREEMPT Thu May 27 14:01:37 BST 2021 aarch64 GNU/Linux
Steps to reproduce
See background.
Expected behaviour
When Peer B force closes on offline peer/forward peer, it should immediately fail the HTLC backward to prevent peer A force close.
Actual behaviour
Cascade of Force Close down the chain.
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 4
- Comments: 29 (6 by maintainers)
Given the setup,
Peer Alice -> Peer Bob -> Offline Peer Charlie, if Charlie is offline during the whole time, then yeah it’s safe to cancel the HTLC, but you can’t be sure. If Charlie comes online after the FC, there are two scenarios,This means Bob would not lose the HTLC if Charlie decides to come online and claim it for w/e reason. However, if Bob cancels the HTLC with Alice after the FC, he is at risk of losing it if Charlie decides to cheat.
So IMO canceling early is not a good choice. Instead, assuming this is an anchor channel, the most feasible way is to fee bump the force close tx.
Of course. Disconnect and connect the peer again.
I have seen many of these cases where the commitment fee rate is at around 10sat/vbyte (see https://github.com/lightningnetwork/lnd/discussions/8271), although it should be higher (https://github.com/lightningnetwork/lnd/issues/8240#issuecomment-1854546090).
Make sure you reconnect to peer 3 when the htlc approaches the blockdealine + 13 blocks, only then will your peer fail the htlc back and no FC will happen on your channel.
The max-anchor-commitfee is default to 10sats, but I am wondering why the channel is not CPFPed, maybe its already purged out of the mempool by the respetive nodes, then he will not be able to bump it.
@ziggie1984 yes please!
Think we can close this