lnd: LND crashed (panic) during QueryRoutes / SendToRouteSync

Background

I had intense code executions QueryRoutes / SendToRouteSync with big lists of ignored_pairs & ignored_nodes. One from LNDs (lnd-01) was crashed

Your environment

version of lnd - 0.8.0-beta

Steps to reproduce

May be intense rebalancing through QueryRoutes (with ignored_pairs & ignored_nodes) -> SendToRouteSync executions.

stacktrace:

2019-11-12 12:53:08.431 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:08.819 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.160 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.694 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.823 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.453 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.505 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.878 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.944 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.204 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.723 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.912 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.307 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.447 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.921 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.926 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.482 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.620 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.701 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.789 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:14.545 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
panic: runtime error: index out of range [20] with length 20

goroutine 8943415 [running]:
github.com/lightningnetwork/lnd/routing/route.(*Route).ToSphinxPath(0xc00648a140, 0x1da7740, 0xc010fbda00, 0x20)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/route/route.go:306 +0x53a
github.com/lightningnetwork/lnd/routing.generateSphinxPacket(0xc00648a140, 0xc0097def3c, 0x20, 0x20, 0xc01e500030, 0x0, 0xc00bede880, 0xf26542, 0xfc5f00, 0xc0000386f8, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1494 +0x4b
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).createNewPaymentAttempt(0xc0085c4f00, 0x40, 0x40, 0xc001034b80, 0xc01281b278, 0x52e342)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:262 +0x475
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).resumePayment(0xc0085c4f00, 0x0, 0x0, 0x0, 0x0, 0x1, 0xb4e4ac, 0xc000010120)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:52 +0xd0a
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).sendPayment(0xc0004d0140, 0x0, 0xc0097def00, 0x14d6860, 0xc007ab2b40, 0x0, 0x0, 0x0, 0x0, 0x1151b00, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1812 +0x1d7
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).SendToRoute(0xc0004d0140, 0x58ca5384bd9010e2, 0xc79742f1db16e50, 0x64bf8000ad875178, 0x7e996cf4e75018da, 0xc00648a140, 0x0, 0x0, 0x0, 0x0, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1724 +0x20f
github.com/lightningnetwork/lnd.(*rpcServer).dispatchPaymentIntent(0xc0001f8160, 0xc018f88420, 0x0, 0x0, 0x0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3172 +0xac
github.com/lightningnetwork/lnd.(*rpcServer).sendPaymentSync(0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc01fefe200, 0x0, 0xc005b21700, 0x1514d40)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3412 +0x11f
github.com/lightningnetwork/lnd.(*rpcServer).SendToRouteSync(0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc00648a050, 0xc0001f8160, 0xd26625, 0xc001652360)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3387 +0x9c
github.com/lightningnetwork/lnd/lnrpc._Lightning_SendToRouteSync_Handler.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x0, 0x0, 0x10, 0xc001898000)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:11745 +0x89
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x1199ba0, 0xc015510001, 0xc00dd82180, 0xc005b21920)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.0/chain.go:31 +0x114
github.com/lightningnetwork/lnd.errorLogUnaryServerInterceptor.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00648a0a0, 0xc00fa51ba0, 0xc01c86c010, 0xc015510000, 0xc00648a0a0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/log.go:248 +0x81
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x1, 0x1, 0x0, 0x0)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.0/chain.go:34 +0x9c
github.com/lightningnetwork/lnd/macaroons.(*Service).UnaryServerInterceptor.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00648a0a0, 0x0, 0xc001c04100, 0xc005b21a10, 0x40e6a8)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/macaroons/service.go:124 +0x150
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00fa51b60, 0xc0117a3a80, 0x5aa858, 0x11344e0, 0xc007ab2750)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.0/chain.go:39 +0x155
github.com/lightningnetwork/lnd/lnrpc._Lightning_SendToRouteSync_Handler(0x11eec80, 0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc00648a000, 0xc0014dc960, 0x14f5c40, 0xc007ab2750, 0xc01001a000, 0x85d)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:11747 +0x14b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100, 0xc0014dcd50, 0x1d80960, 0x0, 0x0, 0x0)
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:966 +0x46a
google.golang.org/grpc.(*Server).handleStream(0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100, 0x0)
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:1245 +0xd16
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00abd0d70, 0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100)
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:685 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:683 +0xa1

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (1 by maintainers)

Most upvoted comments

Currently, you unfortunately cannot set the max number of hops on a queryroutes call.

But at least you can check whether the route returned from queryroutes is 20 hops. If that is the case, don’t fire SendToRoute if you don’t want to crash the node

joostjager on Nov 13, 2019

Yes, sounds like max hops exceeded. We need to fix that. From the code

// NumMaxHops is the maximum path length. This should be set to an estimate of
// the upper limit of the diameter of the node graph.
//
// TODO(roasbeef): adjust due to var-payloads?
const NumMaxHops = 20

joostjager on Nov 13, 2019