nats-server: Placed an entry at the wrong index

Defect

Versions of nats-server and affected client libraries used:

NATS Server v2.3.2, deployed by Helm chart v0.8.4

OS/Container environment:

RKE 1.2.9, Kubernetes 1.2.80, Docker 20.10.7

Steps or code to reproduce the issue:

It’s a 5-node cluster. The cluster was under use for quite some time before it was restarted with kubectl rollout restart statefulset/nats.

Expected result:

All cluster pods restarted one by one, in the rolling manner.

Actual result:

4 out of 5 pods restart successfully. One goes into crash cycle with the following logging output tail:

[83] 2021/07/31 07:45:33.165840 [DBG] RAFT [cnrtt3eg - C-R3F-er29H0Re] Update peers from leader to map[bkCGheKT:0xc0005ce100 cnrtt3eg:0xc00013f120 yrzKKRBu:0xc0005ce0f0]
panic: [nats-2-S-R3F-VkruMIMt] Placed an entry at the wrong index, ae is &{leader:yrzKKRBu term:34 commit:208 pterm:34 pindex:208 entries: 1}, seq is 218, n.pindex is 208



goroutine 154 [running]:
github.com/nats-io/nats-server/server.(*raft).storeToWAL(0xc000b44000, 0xc07fab7100, 0xe99920, 0x8)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:2774 +0x357
github.com/nats-io/nats-server/server.(*raft).processAppendEntry(0xc000b44000, 0xc07fab7100, 0xc08e06a0c0)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:2636 +0x8c5
github.com/nats-io/nats-server/server.(*raft).runAsFollower(0xc000b44000)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:1464 +0x1ce
github.com/nats-io/nats-server/server.(*raft).run(0xc000b44000)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:1416 +0xed
created by github.com/nats-io/nats-server/server.(*Server).startGoRoutine
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:2827 +0xc5



goroutine 154 [running]:
github.com/nats-io/nats-server/server.(*raft).storeToWAL(0xc000b44000, 0xc07fab7100, 0xe99920, 0x8)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:2774 +0x357
github.com/nats-io/nats-server/server.(*raft).processAppendEntry(0xc000b44000, 0xc07fab7100, 0xc08e06a0c0)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:2636 +0x8c5
github.com/nats-io/nats-server/server.(*raft).runAsFollower(0xc000b44000)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:1464 +0x1ce
github.com/nats-io/nats-server/server.(*raft).run(0xc000b44000)
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/raft.go:1416 +0xed
created by github.com/nats-io/nats-server/server.(*Server).startGoRoutine
	/home/travis/gopath/src/github.com/nats-io/nats-server/server/server.go:2827 +0xc5

Full log