rqlite: rqlited 5.10.2 cluster on k8s crashed

version 5.10.2

5 rqlite servers cluster

env : K8s 1.20

debug output:

            _ _ _
           | (_) |
  _ __ __ _| |_| |_ ___
 | '__/ _  | | | __/ _ \   The lightweight, distributed
 | | | (_| | | | ||  __/   relational database.
 |_|  \__, |_|_|\__\___|
         | |               www.rqlite.com
         |_|

[rqlited] 2021/04/09 07:44:05 rqlited starting, version v5.10.2, commit 125ae547879fc5a5b2cbb672b8f5011c171e5907, branch master
[rqlited] 2021/04/09 07:44:05 go1.15, target architecture is amd64, operating system target is linux
[rqlited] 2021/04/09 07:44:05 launch command: /rqlited -node-id rqlite-3 -raft-snap 4096 -raft-timeout 4s -raft-leader-lease-timeout 3s -raft-election-timeout 5s -http-addr rqlite-3.rqlite:4001 -raft-addr rqlite-3.rqlite:4002 /node
[rqlited] 2021/04/09 07:44:05 no preexisting node state detected in /node, node may be bootstrapping
[rqlited] 2021/04/09 07:44:05 no join addresses set
[store] 2021/04/09 07:44:05 opening store with node ID rqlite-3
[store] 2021/04/09 07:44:05 ensuring directory at /node exists
[store] 2021/04/09 07:44:05 0 preexisting snapshots present
[store] 2021/04/09 07:44:05 first log index: 0, last log index: 0, last command log index: 0:
2021-04-09T07:44:05.861Z [INFO]  raft: initial configuration: index=0 servers=[]
[store] 2021/04/09 07:44:05 executing new cluster bootstrap
2021-04-09T07:44:05.861Z [INFO]  raft: entering follower state: follower="Node at 10.129.6.20:4002 [Follower]" leader=
2021-04-09T07:44:13.849Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader=
2021-04-09T07:44:13.849Z [INFO]  raft: entering candidate state: node="Node at 10.129.6.20:4002 [Candidate]" term=2
2021-04-09T07:44:13.933Z [INFO]  raft: election won: tally=1
2021-04-09T07:44:13.933Z [INFO]  raft: entering leader state: leader="Node at 10.129.6.20:4002 [Leader]"
[store] 2021/04/09 07:44:13 waiting for up to 2m0s for application of initial logs
[http] 2021/04/09 07:44:14 service listening on 10.129.6.20:4001
[rqlited] 2021/04/09 07:44:14 node is ready
[rqlited] 2021/04/09 07:45:00 http: panic serving 10.128.7.123:44710: interface conversion: interface {} is nil, not *store.fsmExecuteResponse
goroutine 530 [running]:
net/http.(*conn).serve.func1(0xc000292460)
	/home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xbf3f80, 0xc0003ddd70)
	/home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
github.com/rqlite/rqlite/store.(*Store).execute(0xc00008da00, 0xc0004f4300, 0x41c878, 0x50, 0xc57ee0, 0xc000298901, 0xc0003506e0)
	/tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:591 +0x4a9
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00008da00, 0xc0004f4300, 0x200, 0xc0002180f8, 0x1, 0x1, 0x0)
	/tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc000280a80, 0xdb1ba0, 0xc0006be2a0, 0xc0004d2500)
	/tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc000280a80, 0xdb1ba0, 0xc0006be2a0, 0xc0004d2500)
	/tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc000280b60, 0xdb1ba0, 0xc0006be2a0, 0xc0004d2500)
	/home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000292460, 0xdb3860, 0xc00019c080)
	/home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
	/home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
panic: failed to unmarshal execute subcommand: unexpected EOF

goroutine 13 [running]:
github.com/rqlite/rqlite/store.(*Store).Apply(0xc00008da00, 0xc000111a68, 0x0, 0x0)
	/tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:1042 +0xc0b
github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc000195c20)
	/tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/raft@v1.2.0/fsm.go:90 +0x2c2
github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc000372200, 0x1, 0x40)
	/tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/raft@v1.2.0/fsm.go:113 +0x75
github.com/hashicorp/raft.(*Raft).runFSM(0xc00028a000)
	/tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/raft@v1.2.0/fsm.go:219 +0x3c4
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc00028a000, 0xc000126c90)
	/tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/raft@v1.2.0/state.go:146 +0x55
created by github.com/hashicorp/raft.(*raftState).goFunc
	/tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/raft@v1.2.0/state.go:144 +0x66

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks so much for your reports on this issue – it helped me find a pretty bad bug. I believe v5.11.1 addresses this issue (all reports you have filed were from panics in the compression/decompression code) so I’ll close this issue. Please re-open if you see more panics.