cometbft: Node panics with SIGSEGV when restoring from an atomic disk snapshot

Bug Report

Setup

CometBFT version (use cometbft version or git rev-parse --verify HEAD if installed from source): 0.37.2

Have you tried the latest version: no (checked, however, if the code has changed)

ABCI app (name for built-in, URL for self-written if it’s publicly available): closed-source (based on kvstore)

Environment:

  • OS (e.g. from /etc/os-release): Linux

What happened?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x160 pc=0xd3b7bf]

(during restoring from an atomic disk snapshot)

What did you expect to happen?

Normal operation.

How to reproduce it

Recover a node from an atomic disk snapshot from the past, e.g., AWS EBS backup.

Logs

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x160 pc=0xd3b7bf]

goroutine 1 [running]:
github.com/cometbft/cometbft/state.execBlockOnProxyApp({0x1321ac8?, 0xc0003dd0a0}, {0x13275e0, 0xc00305a060}, 0x0, {0x1329098, 0xc000010f18}, 0xc00088bd40?)
	github.com/cometbft/cometbft@v0.37.2/state/execution.go:331 +0xdf
github.com/cometbft/cometbft/state.ExecCommitBlock({0x13275e0, 0xc00305a060}, 0x0, {0x1321ac8, 0xc0003dd0a0}, {0x1329098?, 0xc000010f18?}, 0xc0000a80d8?)
	github.com/cometbft/cometbft@v0.37.2/state/execution.go:612 +0x67
github.com/cometbft/cometbft/consensus.(*Handshaker).replayBlocks(_, {{{0xb, 0x1}, {0xc002f5bcd2, 0x6}}, {0xc002f5bcd8, 0x7}, 0x1, 0x255c5e, {{0xc00028b1a0, ...}, ...}, ...}, ...)
	github.com/cometbft/cometbft@v0.37.2/consensus/replay.go:471 +0x248
github.com/cometbft/cometbft/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x1}, {0xc002f5bcd2, 0x6}}, {0xc002f5bcd8, 0x7}, 0x1, 0x255c5e, {{0xc00028b1a0, ...}, ...}, ...}, ...)
	github.com/cometbft/cometbft@v0.37.2/consensus/replay.go:393 +0x45b
github.com/cometbft/cometbft/consensus.(*Handshaker).Handshake(0xc000822c40, {0x132a100, 0xc00089c540})
	github.com/cometbft/cometbft@v0.37.2/consensus/replay.go:268 +0x3ca
github.com/cometbft/cometbft/node.doHandshake({_, _}, {{{0xb, 0x1}, {0xc002f5bcd2, 0x6}}, {0xc002f5bcd8, 0x7}, 0x1, 0x255c5e, ...}, ...)
	github.com/cometbft/cometbft@v0.37.2/node/node.go:328 +0x198
github.com/cometbft/cometbft/node.NewNode(0xc0000d18c0, {0x131d990, 0xc000326be0}, 0xc0003d5760, {0x1316740, 0xc000010dc8}, 0xc0001d1728?, 0xbb3b25?, 0xc0002ed0e0, {0x1321ac8, ...}, ...)
	github.com/cometbft/cometbft@v0.37.2/node/node.go:778 +0x56d

Anything else we need to know

The nil dereference error seems to happen here: https://github.com/cometbft/cometbft/blob/v0.37.2/state/execution.go#L331, which points to block to be the only thing that can be nil (block.Txs). This one gets executed by ExecCommitBlock in the same file, and that one in turn by replay.go.replayBlocks. This one takes the block from the store: h.store.LoadBlock(i).

Note that there are other snapshots (AWS EBS as well as k8s volume ones) that work.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (10 by maintainers)

Most upvoted comments

@cason

It will be interesting to understand what interval of blocks is logged when this issue happens.

{"_msg":"ABCI Handshake App Info","hash":null,"height":2417227,"level":"info","module":"consensus","protocol-version":1,"software-version":"1.0.0","ts":"2024-04-30T11:23:52.096146954Z"}
{"_msg":"ABCI Replay Blocks","appHeight":2417227,"level":"info","module":"consensus","stateHeight":2417231,"storeHeight":2417231,"ts":"2024-04-30T11:23:52.096157724Z"}
{"_msg":"Applying block","height":2417228,"level":"info","module":"consensus","ts":"2024-04-30T11:23:52.096165344Z"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x160 pc=0xd3b7bf]

You mentioned using the kvstore application, which is an example that does not persist anything to disk.

Our application took the persistent version of the kvstore as a template (i.e., we pass the Go LevelDB instance). The application state we store is rather minor. However, we persist the last block height, for example. I will take a closer look at that, thanks.

Thanks @hubert-da for the additional context.

Concerning the context in the issue description in Anything else I should know, if the block.Txs is nil, I don’t think that’s the root cause, it wouldn’t panic.

The idea that it would panic in two places when you check the length (len(block.Txs)) or when you do a range over it (for _, tx := range block.Txs) is not correct, this is safe in Go.

In the length case you get 0 and in the range logic is also safe because it’s nil it’s doesn’t execute the logic within it. You can see a quick example here

But we will investigate further or gather more context from other teams. I’ve checked with some other ops people and they mentioned they tried this in the past and didn’t work too well.

Probably we need to try to reproduce the problem locally so we can do some additional debugging to try to pinpoint the root cause.