tendermint: abci: Tendermint state.AppHash does not match AppHash after replay.

BUG REPORT

Tendermint version :: 692691938ccabc48f5bc312bb37f820a66634ec3

ABCI app :: self-written, not publicly available

Essentially, this is a Java app similar to the Go Counter app. In the context of the error, what it does is the following:

1. Stores the received hash value received in callback requestBeginBlock
2. Stores the received height value received in callback requestEndBlock
3. Returns those two stored values via callback requestInfo

Merkleeyes version :: n/a

Environment

• OS :: Any
• Install tools :: Tendermint from source

What happened

I’m running tendermint node. After killing the process and starting it again, I’m seeing the following error.

$ tendermint node
NOTE[03-04|20:47:24] Loaded PrivValidator                     module=types file=C:\\home\\tglaeser/.tendermint/priv_validator.json privValidator="PrivValidator{642FE53D7B76C5E81F5CEDDA3E8736FFB39B97F3 LH:0, LR:0, LS:0}"
NOTE[03-04|20:47:24] ABCI Handshake                           module=consensus appHeight=0 appHash=
NOTE[03-04|20:47:24] ABCI Replay Blocks                       module=consensus appHeight=0 storeHeight=0 stateHeight=0
NOTE[03-04|20:47:24] ConsensusReactor                         module=consensus fastSync=false
NOTE[03-04|20:47:24] Catchup by replaying consensus messages  module=consensus height=1
NOTE[03-04|20:47:24] Replay: Done                             module=consensus
NOTE[03-04|20:47:24] Starting RPC HTTP server on tcp socket 0.0.0.0:46657 module=rpcserver
NOTE[03-04|20:47:24] Started node                             module=main nodeInfo="&{PubKey:PubKeyEd25519{E6D544AF9ADF09C0B989FD828AF1EB309E885A2E416327D5C633820F1DD169B5} Moniker:anonymous Network:test-chain-9Rawjq RemoteAddr: ListenAddr:70.174.164.77:46656 Version:0.8.0 Other:[wire_version=0.6.0 p2p_version=0.3.5 consensus_version=v1/0.2.2 rpc_version=0.6.0/3 rpc_addr=tcp://0.0.0.0:46657]}"
NOTE[03-04|20:47:25] enterNewRound(1/0). Current: 1/0/RoundStepNewHeight module=consensus
NOTE[03-04|20:47:25] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=03FCA8088B6377DE1844A4ED005941B1505678F2
NOTE[03-04|20:47:25] Finalizing commit of block with 0 txs    module=consensus height=1 hash=03FCA8088B6377DE1844A4ED005941B1505678F2 root=
NOTE[03-04|20:47:26] enterNewRound(2/0). Current: 2/0/RoundStepNewHeight module=consensus
NOTE[03-04|20:47:26] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=CC3786BB7666B5BB5D5B47EE9BC35DC466FB8518
NOTE[03-04|20:47:26] Finalizing commit of block with 0 txs    module=consensus height=2 hash=CC3786BB7666B5BB5D5B47EE9BC35DC466FB8518 root=
NOTE[03-04|20:47:27] enterNewRound(3/0). Current: 3/0/RoundStepNewHeight module=consensus
NOTE[03-04|20:47:27] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=BBC2AFD674CE6CB652E4F4BE9C6C3DCA1BC15524
NOTE[03-04|20:47:28] Finalizing commit of block with 0 txs    module=consensus height=3 hash=BBC2AFD674CE6CB652E4F4BE9C6C3DCA1BC15524 root=
NOTE[03-04|20:47:29] enterNewRound(4/0). Current: 4/0/RoundStepNewHeight module=consensus
NOTE[03-04|20:47:29] enterPrecommit: +2/3 prevoted proposal block. Locking module=consensus hash=07D539BBC5E0E62DC951D810B5D826AD204AE902
NOTE[03-04|20:47:29] Finalizing commit of block with 0 txs    module=consensus height=4 hash=07D539BBC5E0E62DC951D810B5D826AD204AE902 root=
captured interrupt, exiting...
NOTE[03-04|20:47:29] Stopping Node                            module=node

$ tendermint node
NOTE[03-04|20:47:30] Loaded PrivValidator                     module=types file=C:\\home\\tglaeser/.tendermint/priv_validator.json privValidator="PrivValidator{642FE53D7B76C5E81F5CEDDA3E8736FFB39B97F3 LH:4, LR:0, LS:3}"
NOTE[03-04|20:47:30] ABCI Handshake                           module=consensus appHeight=4 appHash=07D539BBC5E0E62DC951D810B5D826AD204AE902
NOTE[03-04|20:47:30] ABCI Replay Blocks                       module=consensus appHeight=4 storeHeight=4 stateHeight=4
panic: Tendermint state.AppHash does not match AppHash after replay. Got 07D539BBC5E0E62DC951D810B5D826AD204AE902, expected

goroutine 1 [running]:
github.com/tendermint/tendermint/consensus.(*Handshaker).checkAppHash(0xc0428b47b0, 0xc042456620, 0x14, 0x14, 0x6, 0x6)
        C:/home/projects/go/src/github.com/tendermint/tendermint/consensus/replay.go:340 +0x26d
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc0428b47b0, 0xc042456620, 0x14, 0x14, 0x4, 0xf18860, 0xc042464360, 0x14, 0x14, 0x0, ...)
        C:/home/projects/go/src/github.com/tendermint/tendermint/consensus/replay.go:255 +0x408
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc0428b47b0, 0xf18860, 0xc042464360, 0x0, 0x0)
        C:/home/projects/go/src/github.com/tendermint/tendermint/consensus/replay.go:209 +0x458
github.com/tendermint/tendermint/proxy.(*multiAppConn).OnStart(0xc042464360, 0xc042127880, 0x15)
        C:/home/projects/go/src/github.com/tendermint/tendermint/proxy/multi_app_conn.go:100 +0x219
github.com/tendermint/tendermint/vendor/github.com/tendermint/go-common.(*BaseService).Start(0xc042464360, 0x0, 0x0, 0x0)
        C:/home/projects/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/go-common/service.go:96 +0x2c9
github.com/tendermint/tendermint/node.NewNode(0xf195a0, 0xc04246fe80, 0xc042133860, 0xf0b7c0, 0xc042461b30, 0x0)
        C:/home/projects/go/src/github.com/tendermint/tendermint/node/node.go:73 +0x599
github.com/tendermint/tendermint/node.NewNodeDefault(0xf195a0, 0xc04246fe80, 0x1)
        C:/home/projects/go/src/github.com/tendermint/tendermint/node/node.go:53 +0xc1
main.run_node(0xf195a0, 0xc04246fe80)
        C:/home/projects/go/src/github.com/tendermint/tendermint/cmd/tendermint/run_node.go:49 +0x58d
main.main()
        C:/home/projects/go/src/github.com/tendermint/tendermint/cmd/tendermint/main.go:46 +0x2c0

What you expected to happen:

Well, it seems that appHeight == storeHeight == stateHeight and appHash == hash for the last finalized block, so it is not clear why the Tendermint node cannot resume.

How to reproduce it:

Just testing the same app after #409 was fixed.