lnd: [bug]: LND can't connect to peers/channels, sync to chain is lost

Thanks! Would be nice to get a confirmation that the issue is fixed by #8024. A faster filesystem/disks could make a multi-threading issue less likely to occur.

@Reckless-Satoshi, did #8024 fix the issue?

Since we changed the file system for a much faster one where LND is the only service, the hangs of lncli addinvoice have not happened again. It has been over a week without a crash of that kind (this is still on v0.17.0-rc2).

However, we had 3 lncli getinfo hangs in the same week. We are not running yet the fix in #8024. I collected goroutine?debug=2 while lncli getinfo was not responsive in case it is useful I could share it.

Summarized my investigation and current conclusions here: https://github.com/lightningnetwork/lnd/issues/8023.

@Reckless-Satoshi thanks for that last trace! That was exactly what we needed to get to the bottom of this. I think we have the root cause, and are discussing the best route to resolve the issue (have a few candidates of varying diff size).

@Reckless-Satoshi worth noting that for the blocking/mutex profiles, you only want to enable them when you want a profile, then turn it off. Otherwise it adds extra load since it continues to collect samples unlike the other profiling options.

Thank you! By the way, GitHub also allows uploads of text files, so I’m re-posting the full, combined file here again: goroutinedump.txt

The full goroutine dump was very helpful! It looks like we have a deadlock in the HODL invoice system:

--> This goroutine is trying to send on the hodlChannel, which is buffered but probably already has a message in it. This is holding the invoice registry main lock.
goroutine 1667988 [select, 32 minutes]:
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).notifyHodlSubscribers(0xc0005d83c0, {0x22ca860, 0xc015d33e20})
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:1714 +0x265
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).cancelInvoiceImpl(0xc0005d83c0, {0x4a, 0xe, 0x35, 0x49, 0xd6, 0x77, 0xec, 0xdc, 0x86, ...}, ...)
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:1371 +0x585
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).CancelInvoice(...)
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:1293
github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.(*Server).CancelInvoice(0xc003f0db40, {0x1cbd1c0?, 0x1dc2da0?}, 0x0?)
	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_server.go:309 +0xa5
.....


--> 108 goroutines are attempting to add invoices but are blocked on the main invoice registry lock:

goroutine 1690726 [sync.Mutex.Lock, 11 minutes]:
sync.runtime_SemacquireMutex(0x17ffffffffffffff?, 0x0?, 0x1f0000c017c0d050?)
	runtime/sema.go:77 +0x26
sync.(*Mutex).lockSlow(0xc0005d83c0)
	sync/mutex.go:171 +0x165
sync.(*Mutex).Lock(...)
	sync/mutex.go:90
sync.(*RWMutex).Lock(0xc017c0d0a8?)
	sync/rwmutex.go:147 +0x36
github.com/lightningnetwork/lnd/invoices.(*InvoiceRegistry).AddInvoice(0xc0005d83c0, 0xc0253b6c40, {0x4a, 0x4f, 0x96, 0x4a, 0x90, 0x63, 0x96, 0xf8, ...})
	github.com/lightningnetwork/lnd/invoices/invoiceregistry.go:584 +0x3d
github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.AddInvoice({0x0?, 0x0?}, 0xc017c0d538, 0xc017c0d588)
	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/addinvoice.go:481 +0x1506
github.com/lightningnetwork/lnd/lnrpc/invoicesrpc.(*Server).AddHoldInvoice(0xc003f0db40, {0x22e0938, 0xc01060e3f0}, 0xc028838780)
	github.com/lightningnetwork/lnd/lnrpc/invoicesrpc/invoices_server.go:367 +0x36c

I’ll see if I can come up with a fix. If there’s any way for you to temporarily stop using HODL invoices then that should unclog the node. But I assume that’s a main part of your application… So hopefully we can get a patch out soon.

Those stack traces were super helpful! We have a candidate fix here: https://github.com/lightningnetwork/lnd/issues/7928

Sent new logs with Debug verbosity. This time it seems something did fail before any peer was connected.

Edit: sent another run where it connected to 2 channels before chain sync went false.

hieblmi at lightning dot engineering

Hi, thanks for reporting the issue. Did this behavior occur the first time after you adjusted settings or did it start without intervention? Would you be able to provide lnd logs that contain the start up as well as showing the decay of active peers? Can also mail the logs if you don’t want to attach them here.