lnd: [bug]: unable to forward, wrong "insufficient bandwidth"

Background

I have more than enough liquidity in one of my channels and should be able to serve outgoing forward requests. However, this doesn’t work. Instead, lnd reports “insufficient bandwidth”. Details below.

Your environment

  • lnd v0.15.4-beta
  • Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
  • bitcoind v23

Steps to reproduce

Get forwarding request out via channel X.

Expected behaviour

Forwarding request is served, fees are earned, local balance is reduced.

Actual behaviour

“insufficient bandwidth to route htlc: x is larger than y” (canSendHtlc in link.go)

First observed failure:

  • Date: 2022-10-25 18:31:17
  • Requested amount: 188,242,405 msat
  • Available (according to lnd): 3,005,999 msat

Balance at 2022-10-25 18:29:25+02:

  • local: 9,897,025 sat
  • local reserve: 100,000 sat
  • remote: 100,078 sat
  • remote reserve: 100,000 sat

Most recent failure:

  • Date: 2022-11-03 13:18:31
  • Requested amount: 50,024,602 msat
  • Available (according to lnd): 2,843,999 msat

Balance at 2022-11-03 13:14:47+01:

  • local: 9,896,568 sat
  • local reserve: 100,000 sat
  • remote: 100,712 sat
  • remote reserve: 100,000 sat

lncli listchannels:

{
            "active": true,
            "remote_pubkey": "03627ebe50fc6eb80b0caab0c3714958c701eda735e3c29588e83150d6d4a93976",
            "channel_point": "db8a9a4d483e0fd0c84e5a004d51dd51ecbb113329729c18db6a5112bab61a4f:0",
            "chan_id": "834535922707529728",
            "capacity": "10000000",
            "local_balance": "9891564",
            "remote_balance": "108241",
            "commit_fee": "195",
            "commit_weight": "724",
            "fee_per_kw": "269",
            "unsettled_balance": "0",
            "total_satoshis_sent": "2036708",
            "total_satoshis_received": "11928272",
            "num_updates": "9950",
            "pending_htlcs": [
            ],
            "csv_delay": 144,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "100000",
            "remote_chan_reserve_sat": "100000",
            "static_remote_key": true,
            "commitment_type": "STATIC_REMOTE_KEY",
            "lifetime": "174093",
            "uptime": "174093",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 144,
                "chan_reserve_sat": "100000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "18446744073709551615",
                "min_htlc_msat": "0",
                "max_accepted_htlcs": 30
            },
            "remote_constraints": {
                "csv_delay": 1201,
                "chan_reserve_sat": "100000",
                "dust_limit_sat": "546",
                "max_pending_amt_msat": "9900000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0"
        }

lncli getchaninfo:

{
    "channel_id": "834535922707529728",
    "chan_point": "db8a9a4d483e0fd0c84e5a004d51dd51ecbb113329729c18db6a5112bab61a4f:0",
    "last_update": 1667403066,
    "node1_pub": "027ce055380348d7812d2ae7745701c9f93e70c1adeb2657f053f91df4f2843c71",
    "node2_pub": "03627ebe50fc6eb80b0caab0c3714958c701eda735e3c29588e83150d6d4a93976",
    "capacity": "10000000",
    "node1_policy": {
        "time_lock_delta": 99,
        "min_htlc": "1",
        "fee_base_msat": "0",
        "fee_rate_milli_msat": "1",
        "disabled": false,
        "max_htlc_msat": "10000000000",
        "last_update": 1667403066
    },
    "node2_policy": {
        "time_lock_delta": 34,
        "min_htlc": "1",
        "fee_base_msat": "0",
        "fee_rate_milli_msat": "135",
        "disabled": false,
        "max_htlc_msat": "104096134",
        "last_update": 1667143395
    }
}

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (17 by maintainers)

Commits related to this issue

Most upvoted comments

I now have the same issue with at least five (!) other peers, all of them are anchor channels, though! I managed to send out 10k sat via rebalance-lnd in all cases.

Other users are affected, too. I received one report via Twitter and two via Telegram. In all cases, the situations matches mine (low remote balance, log message indicating x999msat for values <500sat even though there’s far more on the local side).

Yes, the remote party has a very low balance, especially considering the reserve. However, I don’t understand why this stops me from sending funds to their side, i.e. creating a more balanced channel. It feels like the channel drove itself into a dead end?

So the initiator pays fees for a commitment transaction the entire time. They need to have enough fees to pay for a force close assigned, and also they pay all the fees for a co-op close.

Each time you add an HTLC, they need to have enough funds to pay for that HTLC (it increases the size of the commitment transaction). This is the case even if they aren’t the one adding an HTLC. In the case above, the initiator only has 100,078 sats on their side. The reserve itself is just 78 sats below that at 100k. As as a result, the channel can’t be utilized unless the fee rate (for the commitment) transaction is reduced to the point that it can support another HTLC, then things can be rebalanced somewhat.

The relationship between the fees paid, HTLC weight and the reserve mostly works on the spec level. However, there’re some degenerate cases when a channel isn’t able to be used due to the initiator not being able to pay fees. I think what happened is that they tried to send everything out of the channel, realized the reserve was there, then played around to send the most they could out, leaving that amount.

Returning to the line you linked above: https://github.com/lightningnetwork/lnd/blob/master/lnwallet/channel.go#L7030-L7035

Since theirBalance < htlcCommitFee is true, they don’t have enough fees to pay for the HTLC. The function only gets down here if you aren’t the initiator. In order to ensure we don’t route across this (only hard fails are possible in the protocol when a constraint is violated, you can’t “undo” the HTLC). Then ourBalance >= nonDustHtlcAmt is also true, but we don’t want to allow this to happen (the channel would basically force close), so we say we have one less than the HTLC amount. This then prevents the HTLC from being routed in general.

There is a way to get around this though: send a small amount continually back to yourself as a rebalance. If the amount is below dust, then no HTLC is added, so no extra fees are required.

Prior spec discussion related to this issue can be found here: https://github.com/lightning/bolts/issues/728.

~10% of my channels are presently unusable by this bug, or in other terms 60% of my local liquidity (outgoing) is frozen and cannot be rebalanced, forwarded or sent. Issue that supercedes appears to be an inversion of this bug, for low local balances. I have high local balances (98%+) and receive this error, I don’t believe that this issue has been resolved.

0.17.0-beta

I have same issue. HTLC manager is logging bandwidth when LND starts up, and it suddenly becomes a small value.

[INF] HSWC: ChannelLink(0be39eb940989bf68726d08e57adc1b80124b4df6691db9494df691e1a718c95:0): HTLC manager started, bandwidth=353999 mSAT

It has occurred several times, usually at “353999 mSAT”.

No, not according to the logs, and not right now (see lncli listchannels output above). Considering that I had more than 50 failures, this also seems very unlikely.