lnd: lnd-0.14.0-rc1 hangs after a few failed rebalance attempts
Background
Some subsystems of lnd stop working after I run rebalance-lnd and a few rebalance attempts fail (rebalance lnd reports “Temporary payment failure”). In particular, CRTR subsystem stops writing any logs even in debug, and lnd stops responding to lncli - for example lncli getinfo waits indefinitely. And the disc activity of lnd drops significantly, from at least 200 kb/s to some 10 kb/s. But it does not hang completely, it still produces log lines of PEER, DISC, HSWC and SRVR.
The last log line produced by CRTR is
CRTR: Searching for path to 033878501f9a4ce97dba9a6bba4e540eca46cb129a322eb98ea1749ed18ab67735, sending 300000000 mSAT
Tested twice so far, once just few minutes after lnd started and connected to peers, lnd hanged after first rebalance failure. Second time, lnd ran fine for about 10 hours while no rebalancing was done, then I started rebalance-lnd and it hanged after 3 attempts.
Your environment
- version of lnd: 0.14.0-beta.rc1
- db.no-graph-cache NOT set
- which operating system (uname -aon *Nix):Linux ubuntu 5.11.0-1021-raspi #22-Ubuntu SMP PREEMPT Wed Oct 6 17:30:38 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
- version of btcd,bitcoind, or other backend: bitcoin-0.21.1
- version of rebalance-lnd: commit aca3ed199cffa6901b3cd35c8d283e24f68cf259 of 31/oct/2021 it uses grpcio 1.41.1
- libc6:arm64 2.33-0ubuntu5
- Ubuntu 21.04 64-bit, last updated 31/10/2021
- Raspberry Pi 4 with 4 GB RAM
Steps to reproduce
- run lnd and wait until it initializes (reads db, connect to most of peers)
- run rebalance-lnd (https://github.com/C-Otto/rebalance-lnd)
Expected behaviour
lnd should keep working, forwarding, responding to lncli getinfo.
Actual behaviour
lnd hangs, at least CRTR subsystem.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 23 (11 by maintainers)
Should’ve been fixed by #5941, will re-open if we get a repro that indicates otherwise