tendermint: BUG REPORT: Full network crash - panic: notified txs available but mempool is empty!
tendermint version: 0.10.4-'9af837c ABCI App: ethermint 0.4.0
Environment: "16.04.3 LTS (Xenial Xerus)
ethremint logs
ubuntu@ip-172-31-23-39:~$ tail -200f .ethermint/logs/ethermint.log
WARN [11-01|12:28:46] DeliverTx error err="invalid nonce: have 10103484, expected 10103486"
WARN [11-01|12:28:46] DeliverTx error err="invalid nonce: have 10103485, expected 10103486"
WARN [11-01|12:28:51] DeliverTx error err="invalid nonce: have 10103499, expected 10103504"
WARN [11-01|12:28:51] DeliverTx error err="invalid nonce: have 10103500, expected 10103504"
WARN [11-01|12:28:51] DeliverTx error err="invalid nonce: have 10103501, expected 10103504"
WARN [11-01|12:28:51] DeliverTx error err="invalid nonce: have 10103502, expected 10103504"
WARN [11-01|12:28:51] DeliverTx error err="invalid nonce: have 10103503, expected 10103504"
WARN [11-01|12:29:03] DeliverTx error err="invalid nonce: have 10103545, expected 10103552"
WARN [11-01|12:29:03] DeliverTx error err="invalid nonce: have 10103546, expected 10103552"
WARN [11-01|12:29:03] DeliverTx error err="invalid nonce: have 10103547, expected 10103552"
WARN [11-01|12:29:03] DeliverTx error err="invalid nonce: have 10103548, expected 10103552"
WARN [11-01|12:29:03] DeliverTx error err="invalid nonce: have 10103549, expected 10103552"
WARN [11-01|12:29:06] DeliverTx error err="invalid nonce: have 10103552, expected 10103557"
WARN [11-01|12:29:06] DeliverTx error err="invalid nonce: have 10103553, expected 10103557"
WARN [11-01|12:29:09] DeliverTx error err="invalid nonce: have 10103557, expected 10103558"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103559, expected 10103578"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103560, expected 10103578"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103561, expected 10103578"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103562, expected 10103578"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103563, expected 10103578"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103564, expected 10103578"
WARN [11-01|12:29:12] DeliverTx error err="invalid nonce: have 10103565, expected 10103578"
WARN [11-01|12:29:15] DeliverTx error err="invalid nonce: have 10103578, expected 10103589"
WARN [11-01|12:29:15] DeliverTx error err="invalid nonce: have 10103579, expected 10103589"
WARN [11-01|12:29:20] DeliverTx error err="invalid nonce: have 10103589, expected 10103603"
WARN [11-01|12:29:20] DeliverTx error err="invalid nonce: have 10103590, expected 10103603"
WARN [11-01|12:29:29] DeliverTx error err="invalid nonce: have 10103616, expected 10103635"
WARN [11-01|12:29:29] DeliverTx error err="invalid nonce: have 10103617, expected 10103635"
WARN [11-01|12:29:29] DeliverTx error err="invalid nonce: have 10103618, expected 10103635"
WARN [11-01|12:29:29] DeliverTx error err="invalid nonce: have 10103619, expected 10103635"
WARN [11-01|12:29:32] DeliverTx error err="invalid nonce: have 10103635, expected 10103649"
ERROR[11-01|12:33:16] Connection error module=abci-server error=EOF
ERROR[11-01|12:33:16] Connection error module=abci-server error=EOF
ERROR[11-01|12:33:16] Connection error module=abci-server error=EOF
all network nodes crashed with
I[11-01|12:33:16.865] Committed state module=state height=7887791 txs=11 hash=DA00D930AA7008A06C9ED7623EB8F57873FE1FDE75E166F580C9E0E74340C2E5
panic: notified txs available but mempool is empty!
goroutine 34 [running]:
github.com/tendermint/tendermint/mempool.(*Mempool).notifyTxsAvailable(0xc4204b6880)
/go/src/github.com/tendermint/tendermint/mempool/mempool.go:288 +0xd5
github.com/tendermint/tendermint/mempool.(*Mempool).resCbRecheck(0xc4204b6880, 0xc4236d6820, 0xc4236d7110)
/go/src/github.com/tendermint/tendermint/mempool/mempool.go:272 +0x1bc
github.com/tendermint/tendermint/mempool.(*Mempool).resCb(0xc4204b6880, 0xc4236d6820, 0xc4236d7110)
/go/src/github.com/tendermint/tendermint/mempool/mempool.go:212 +0x48
github.com/tendermint/tendermint/mempool.(*Mempool).(github.com/tendermint/tendermint/mempool.resCb)-fm(0xc4236d6820, 0xc4236d7110)
/go/src/github.com/tendermint/tendermint/mempool/mempool.go:95 +0x3e
github.com/tendermint/tendermint/vendor/github.com/tendermint/abci/client.(*socketClient).didRecvResponse(0xc42042ac80, 0xc4236d7110, 0x0, 0x0)
/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/abci/client/socket_client.go:218 +0x261
github.com/tendermint/tendermint/vendor/github.com/tendermint/abci/client.(*socketClient).recvResponseRoutine(0xc42042ac80, 0x105f7e0, 0xc42000c2b0)
/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/abci/client/socket_client.go:177 +0x169
created by github.com/tendermint/tendermint/vendor/github.com/tendermint/abci/client.(*socketClient).OnStart
/go/src/github.com/tendermint/tendermint/vendor/github.com/tendermint/abci/client/socket_client.go:78 +0x1da
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 16 (9 by maintainers)
Commits related to this issue
- Merge pull request #802 from celestiaorg/evan/backport-old-v0.35.x-changes Backport v0.35.x changes to v0.34.x — committed to cmwaters/tendermint by evan-forbes 2 years ago
👍 you may close this one @ebuchman @odeke-em thx for quick fix / release @melekes it was all I have