lnd: lnd 0.9.1 crashing on raspberry pi

Background

lnd crashes after a few minutes for some reason. This happens since 0.9.1. After downgrading to 0.9.0 lnd runs again without any crashes

Your environment

lnd 0.9.1
Raspbian Buster 10.0 (with alle new updates)
bitcoind v0.19.0.1
Raspberry Pi 4 (4GB RAM) with healthy hardware

Steps to reproduce

Update lnd to 0.9.1

Expected behaviour

It should not crash 😃

Actual behaviour

It crashes after a few minutes

log

March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.928 [INF] HSWC: Removing channel link with ChannelID(a4315f6bd5d505247a3380d6e57c80b5057e51dcebae5e494edb609a882d7841)
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.928 [INF] HSWC: ChannelLink(611962:2124:1): starting
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.929 [INF] HSWC: Trimming open circuits for chan_id=611962:2124:1, start_htlc_id=8
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.930 [INF] HSWC: Adding live link chan_id=a4315f6bd5d505247a3380d6e57c80b5057e51dcebae5e494edb609a882d7841, short_chan_id=611962:2124:1
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.930 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(40782d889a60db4e495eaeebdc517e05b5807ce5d680337a2405d5d56b5f31a4:1)
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.930 [INF] HSWC: ChannelLink(611962:2124:1): HTLC manager started, bandwidth=921523000 mSAT
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.930 [INF] SRVR: Negotiated chan series queries with 0216fcf031519e0924e5e4b64a731b207cccb85b13caebffd0f664a32a39bcd368
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.930 [INF] DISC: Creating new GossipSyncer for peer=0216fcf031519e0924e5e4b64a731b207cccb85b13caebffd0f664a32a39bcd368
March 05 10:37:19 xxxxxxx lnd[17811]: 2020-03-05 10:37:19.930 [INF] HSWC: ChannelLink(611962:2124:1): attempting to re-resynchronize
March 05 10:37:20 xxxxxxx lnd[17811]: 2020-03-05 10:37:20.128 [INF] HSWC: ChannelLink(611960:718:1): received re-establishment message from remote side
March 05 10:37:20 xxxxxxx lnd[17811]: 2020-03-05 10:37:20.178 [INF] HSWC: ChannelLink(611962:2124:1): received re-establishment message from remote side
March 05 10:37:22 xxxxxxx lnd[17811]: 2020-03-05 10:37:22.858 [INF] DISC: GossipSyncer(03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601): fetching chan anns for 100 chans
March 05 10:37:23 xxxxxxx lnd[17811]: 2020-03-05 10:37:23.274 [INF] DISC: GossipSyncer(03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601): rate limiting gossip replies, responding in 4.583495212s
March 05 10:37:27 xxxxxxx lnd[17811]: 2020-03-05 10:37:27.858 [INF] DISC: GossipSyncer(03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601): fetching chan anns for 100 chans
March 05 10:37:28 xxxxxxx systemd[1]: lnd.service: Main process exited, code=killed, status=11/SEGV
March 05 10:37:28 xxxxxxx systemd[1]: lnd.service: Failed with result 'signal'.
March 05 10:38:28 xxxxxxx systemd[1]: lnd.service: Service RestartSec=1min expired, scheduling restart.
March 05 10:38:28 xxxxxxx systemd[1]: lnd.service: Scheduled restart job, restart counter is at 2.
March 05 10:38:28 xxxxxxx systemd[1]: Stopped LND Lightning Daemon.
March 05 10:38:28 xxxxxxx systemd[1]: Started LND Lightning Daemon.

lnd.conf

[Application Options]
debuglevel=info
maxpendingchannels=5
alias=...
color=...
rpclisten=0.0.0.0:10009
tlsextraip=...
nat=true
accept-keysend=true

[Bitcoin]
bitcoin.active=1
bitcoin.node=bitcoind
bitcoin.mainnet=1

[autopilot]
autopilot.active=1
autopilot.maxchannels=5
autopilot.allocation=0.8

bitcoin.conf

server=1
daemon=1
upnp=1
txindex=1
rpcuser=...
rpcpassword=...
onlynet=ipv4
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 21 (15 by maintainers)

Commits related to this issue

build: update to x/crypto version w/o broken poly1305 arm assembly In this commit, we update to a new version of `x/crypto` that drops broken ARM assembly that can cause a segfault in systems like ra... — committed to Roasbeef/lnd by Roasbeef 4 years ago
build: update to x/crypto version w/o broken poly1305 arm assembly In this commit, we update to a new version of `x/crypto` that drops broken ARM assembly that can cause a segfault in systems like ra... — committed to Roasbeef/lnd by Roasbeef 4 years ago
build: update to x/crypto version w/o broken poly1305 arm assembly In this commit, we update to a new version of `x/crypto` that drops broken ARM assembly that can cause a segfault in systems like ra... — committed to lightningnetwork/lnd by Roasbeef 4 years ago
build: update to x/crypto version w/o broken poly1305 arm assembly In this commit, we update to a new version of `x/crypto` that drops broken ARM assembly that can cause a segfault in systems like ra... — committed to lightningnetwork/lnd by Roasbeef 4 years ago

Most upvoted comments

@openoms dope! If over the next day or so, we don’t get a conflicting response, we’ll cut v0.9.2, which is the same as v0.9.1, but just includes that commit I referenced.

Roasbeef on Mar 11, 2020

ok, running with the commit b96de11

Will report if still up after a day.

Had no issues since with the updated library, the problem is likely solved, since it is the longest uptime with lnd v0.9.1.

Folks running a RaspiBlitz setup can use this script to test: https://github.com/openoms/lightning-node-management/tree/master/lnd.updates#build-lnd-from-source

Paste the commit hash:
b96de11678f6395b2e7d523af8ec4129e0d21df2

openoms on Mar 10, 2020

ok, running with the commit https://github.com/lightningnetwork/lnd/commit/b96de11678f6395b2e7d523af8ec4129e0d21df2

Will report if still up after a day.

openoms on Mar 9, 2020

We’ve tagged the new version: https://github.com/lightningnetwork/lnd/releases/tag/v0.9.2-beta

Binaries are uploading as I type this, will then have the usual release notes template albeit it’ll be much smaller (only a single change).

Roasbeef on Mar 13, 2020

After testing 0.9.1 built with go1.13 I figured out that no crashes appear anymore. I’m not sure what the problem is but I can confirm that a binary built with go1.13 works like a charm. Tested on armv7 (raspberry pi).

turcane on Mar 5, 2020

Can y’all try out this branch to see if it resolves things? https://github.com/lightningnetwork/lnd/tree/v0.9.2-beta-rc1-branch

It includes one commit on top of v0.9.1 should resolve the issue (there was some old broken ARM assembly for poly1305 that seems to be more of an issue in v0.9.1 for some reason).

Roasbeef on Mar 9, 2020

bingo. this was the problem. after more than 1 hour runtime on two raspberries no crashes anymore. great job!

turcane on Mar 5, 2020