lnd: Still cannot force close inactive channels

To all the channels we lost…

I’m finally closing this issue as the remaining 3.7 BTC have been restored. This is the tool I created to assist in the recovery, combined with SCB of course.

These are the most important things we learned from this issue (for people coming across this when searching):

• SCB restore has a few bugs. 0.8.2 will fix those.
• 400 channels is a lot! lnd comes to its limits when doing bulk operations on that many channels. Commands like closeallchannels or restorechanbackup can take hours (!). And aborting them will most certainly leave your node in an inconsistent state. Always check the log for activity before hitting Ctrl+C on a long-running command. And we probably should add more warnings to lncli.
• Restoring from SCB takes a long time with more than a few dozen channels. Think days or even a week. For example, adding an SCB with 430 channels took about 2.5 hours (just the restorechanbackup command) then the node did chain re-scans for about 3 days (on a fast machine with SSD) then took another 2 days to communicate with the peers to close the channels and sweep the funds.
• SCBs don’t work with offline peers. Repeat after me: Close channels with offline peers! See this article for more information. In this case, there was almost 1 BTC in zombie channels and that would have been lost if @MasterNeuron didn’t keep copies of the channel.db around. Therefore making file based backups can help as long as you know what you are doing and don’t just put the files back in lnd’s data directory. Do SCB for the online channels first, then use my tool to force-close zombie channels (which can still be very risky).
• It’s getting harder to actually lose coins, as this issue shows. But lnd is still in beta! That means it’s far from “it just works and I don’t have to know anything about it”. There’s still a lot of safety code that has yet to be written (I got some usable ideas from working on this issue). So please if you’ve got a life-altering amount of BTC in lnd (which you SHOULDN’T), at least educate yourself about the DOs and DON’Ts. I’m going to write some more explicit documentation on this too.

To all the channels we lost…

I have more than 4 BTC there which is ~30K $… that’s so unserious. I’ve just force-closed my channels and lost them!!!

This means that lnd is trying to connect to the remote peer and ask it for the information it needs to recover the funds from the backup. Maybe the peers of the channels listed aren’t currently online. Can you check if you can connect to them? Pick a channel from the list, find out what peer it is (maybe with the help of 1ml.com) and try lncli connect.

The logs contain commit points, such as in vegardengen’s comment https://github.com/lightningnetwork/lnd/issues/2468#issuecomment-549677751

As a last resort, could we derive the private key for each utxo with MasterNeuron’s lnd aezeed private key and the commit point (if found in the log)? The channel point is right there as well so we can find the relevant utxo with MasterNeuron’s funds.

What I meant when I said, “looks like it is progressing nicely”, is this:

2019-10-31 12:32:27.931 [INF] LTND: Received interrupt 2019-10-31 12:32:27.932 [INF] LTND: Shutting down... 2019-10-31 12:32:27.932 [INF] LTND: Gracefully shutting down. 2019-10-31 12:32:28.150 [ERR] NTFN: Rescan to determine the spend details of outpoint=bbc18a3bda9d24f99b74bdc89b4 d52671365dd9265f3cb770eb27d742a527e5c:0 within range 566116-601590 failed: chain notifier shutting down 2019-10-31 12:32:30.009 [ERR] NTFN: Rescan to determine the spend details of outpoint=30a6add708ccc85aa38c5015029 b23487f6b46d3adf21d974b87260387362097:1 within range 568399-601613 failed: chain notifier shutting down

If you let these run, there’s all reason to believe that your node will find spend details of those outpoints, and it has the keys to sweep them.

I believe this is likely to bring back at least quite a lot of your funds, @guggero has promised to write a script which is likely to tell us to which extent.

The reason why your restore of the static channel backup doesn’t work is something else which we don’t fully understand. Maybe if you could show us the exact steps you tried when restoring it?

For the record, the starting procedure for using a static channel backup should, in your case, be an empty node. No wallet, no channel database, just your 24 word seed and the static channel backup.

But I believe the best way forward is to continue the attempt which you seemed to stop at november 5th. Let it run - for a week or two if it needs to. There’s a lot of scanning your node has to do.

And watch your logs, your listchaintxns and closedchannels and see what happens, if you like. Because every now and then, it should find a closing transaction that it need to sweep funds from.

Looking at the log file, you need to let your node where you started the rescue run for a few days. When you shut it down on October 31, there were a lot of messages like:

Rescan to determine the spend details of outpoint=6a60b1f49b94f31dc4409f210d86aac32454d344466ae3d92f4477c87b7a318d:1 within range 570450-601598 failed: chain notifier shutting down

Which means, the node was still rescanning channels from the chain backend (bitcoind). This can take a long time!

It’s full of errors.

Yes, you are right. That’s why there still is a -beta suffix in each version that we release.

But I am working on fixing those SCB errors. The a height hint greater than 0 must be provided should be an easy fix. I think it happens when there is a channel in the SCB file that was not confirmed yet at the time of the file update. We should just skip this channel instead of failing the whole file.

The message authentication failed is harder to pin down. It either means that you are trying to use a channel backup file that was not created with the node/seed you are trying to restore. But it could also mean that the SCB file is corrupt for some reason.

I’m going to write a script that goes through all your 400 channels and looks at the TX outputs if there is still something to be claimed. This should also give us a final account of how much was actually lost because of justice transactions (if any).

instead of restorechanbackup, you can also use verifychanbackup .

And as for which SCB I’d use - I think I’d prefer to use the last good one before october 2nd .

I’m a bit confused. 3 days ago, you posted a long mail, claiming that your small node had nodeID: 0356c02ffe265f8ff38471e901785528d3da0668cd0a590ed67ee809d929c9bfd4

and the large one: 02725e5abcbf5550fc29e6b19706a1377f25d2b1502684f9be9965b6deac167520

Then you post a picture of something you claim is your upgraded small node, but which have the node ID you previously claimed is the large one.

That’s why I am asking.

@MasterNeuron: your “small” node is now ready to begin the SCB recovery. Please run lncli restorechanbackup --multi_file path/to/channel.backup where channel.backup is the binary backup file that is created by lnd in the .lnd/data/chain/bitcoin/mainnet folder. Use the most recent that you have.

It is normal that the name of the node doesn’t update as long as you don’t have any channels. Without any channels, no broadcasts happen and your node is basically “invisible” to the network.

Yeah I don’t think all their funds are gone. In their past screenshot they have 0 peers, which means there’s something else going on there. Things don’t quite add up, and if we could get some more information or logs @MasterNeuron we’ll actually be able to help.

@MasterNeuron can you DM me at the community Slack? I have a branch up for fund recovery. I’m going to try to help you get our BTC back.

So it makes no sense to make backups. When something happens, you always restore an old state (backups are at least some hours old). When nodes close channels when your node is offline, and this is the case when you restore the backup, then those funds are lost.