rqlite: Getting database disk image is malformed after rqlite interruption
What version are you running? v8.15.2
Are you using Docker or Kubernetes to run your system? No
Are you running a single node or a cluster? Cluster
Issue Description I’m facing an issue where rqlite might fall into “failed to open store: failed to create snapshot store: check failed: journal mode to WAL: database disk image is malformed” after being previously interrupted (killed).
Cluster information (2-nodes cluster): - Node ID: 192.168.15.111, Suffrage=Voter - Node ID: 192.168.15.44, Suffrage=NonVoter - Node Interrumpted: 192.168.15.111, Suffrage=Voter
Launch commands: 192.168.15.111(Voter): rqlited.exe -raft-log-level=DEBUG -node-id 192.168.15.111 -http-addr 192.168.15.111:8906 -raft-addr 192.168.15.111:8904 -join 192.168.15.111:8904 -bootstrap-expect 1 -raft-non-voter=false -node-ca-cert DatabaseCertificateAuthority.cer -node-cert DatabasePublicCertificate.cer -node-verify-client -node-key DatabasePrivateCertificate.pem -http-ca-cert DatabaseCertificateAuthority.cer -http-cert DatabasePublicCertificate.cer -http-verify-client -http-key DatabasePrivateCertificate.pem Local 192.168.15.44 (NonVoter): rqlited.exe -raft-log-level=DEBUG -node-id 192.168.15.44 -http-addr 192.168.15.44:8906 -raft-addr 192.168.15.44:8904 -join 192.168.15.111:8904 -raft-non-voter=true -node-ca-cert DatabaseCertificateAuthority.cer -node-cert DatabasePublicCertificate.cer -node-verify-client -node-key DatabasePrivateCertificate.pem -http-ca-cert DatabaseCertificateAuthority.cer -http-cert DatabasePublicCertificate.cer -http-verify-client -http-key DatabasePrivateCertificate.pem Local
After interrumption, if I start the cluster that’s what happens in the voter machine:
_ _ _
| (_) |
_ __ __ _| |_| |_ ___
| '__/ _ | | | __/ _ \ The lightweight, distributed
| | | (_| | | | || __/ relational database.
|_| \__, |_|_|\__\___|
| | www.rqlite.io
|_|
[rqlited] 2024/01/09 08:35:56 rqlited starting, version 8, SQLite 3.44.0, commit unknown, branch unknown, compiler gc
[rqlited] 2024/01/09 08:35:56 go1.20.2, target architecture is amd64, operating system target is windows
[rqlited] 2024/01/09 08:35:56 launch command: rqlited.exe -raft-log-level=DEBUG -node-id 192.168.15.111 -http-addr 192.168.15.111:8906 -raft-addr 192.168.15.111:8904 -join 192.168.15.111:8904 -bootstrap-expect 1 -raft-non-voter=false -node-ca-cert DatabaseCertificateAuthority.cer -node-cert DatabasePublicCertificate.cer -node-verify-client -node-key DatabasePrivateCertificate.pem -http-ca-cert DatabaseCertificateAuthority.cer -http-cert DatabasePublicCertificate.cer -http-verify-client -http-key DatabasePrivateCertificate.pem Local
[rqlited] 2024/01/09 08:35:56 enabling node-to-node encryption with cert: DatabasePublicCertificate.cer, key: DatabasePrivateCertificate.pem, CA cert DatabaseCertificateAuthority.cer, mutual TLS enabled
[rqlited] 2024/01/09 08:35:56 Raft TCP mux Listener registered with byte header 1
[mux] 2024/01/09 08:35:56 TLS mux serving on 192.168.15.111:8904, advertising 192.168.15.111:8904
[rqlited] 2024/01/09 08:35:56 preexisting node state detected in Local
[cluster] 2024/01/09 08:35:56 service listening on 192.168.15.111:8904
[rqlited] 2024/01/09 08:35:56 cluster TCP mux Listener registered with byte header 2
[http] 2024/01/09 08:35:56 secure HTTPS server enabled with cert DatabasePublicCertificate.cer, key DatabasePrivateCertificate.pem, CA cert DatabaseCertificateAuthority.cer, mutual TLS enabled
[http] 2024/01/09 08:35:56 execute queue processing started with capacity 1024, batch size 128, timeout 50ms
[http] 2024/01/09 08:35:56 service listening on 192.168.15.111:8906
[rqlited] 2024/01/09 08:35:56 HTTP server started
[store] 2024/01/09 08:35:56 opening store with node ID 192.168.15.111, listening on 192.168.15.111:8904
[store] 2024/01/09 08:35:56 ensuring data directory exists at Local
[store] 2024/01/09 08:35:56 old snapshot directory Local\snapshots does not exist, nothing to upgrade
[snapshot-store] 2024/01/09 08:35:56 store initialized using Local\rsnapshots
[snapshot-store] 2024/01/09 08:35:56 checking consistency of snapshot store at Local\rsnapshots
[snapshot-store] 2024/01/09 08:35:56 check complete
[store] 2024/01/09 08:35:56 1 preexisting snapshots present
[store] 2024/01/09 08:35:56 first log index: 1, last log index: 246, last applied index: 246, last command log index: 246:
[store] 2024/01/09 08:35:56 created on-disk database at open
2024-01-09T08:35:56.318-0300 [INFO] raft: starting restore from snapshot: id=15-224-1704799965251 last-index=224 last-term=15 size-in-bytes=134144
[store] 2024/01/09 08:35:56 initiating node restore on node ID 192.168.15.111
[store] 2024/01/09 08:35:56 successfully opened database at Local\db.sqlite due to restore
[store] 2024/01/09 08:35:56 node restored in 7.4983ms
2024-01-09T08:35:56.326-0300 [INFO] raft: snapshot restore progress: id=15-224-1704799965251 last-index=224 last-term=15 size-in-bytes=134144 read-bytes=134144 percent-complete="100.00%"
2024-01-09T08:35:56.326-0300 [INFO] raft: restored from snapshot: id=15-224-1704799965251 last-index=224 last-term=15 size-in-bytes=134144
2024-01-09T08:35:56.326-0300 [INFO] raft: initial configuration: index=3 servers="[{Suffrage:Voter ID:192.168.15.111 Address:192.168.15.111:8904} {Suffrage:Nonvoter ID:192.168.15.44 Address:192.168.15.44:8904}]"
[rqlited] 2024/01/09 08:35:57 checking that any supplied join addresses don't serve HTTP(S)
2024-01-09T08:35:57.220-0300 [INFO] raft: entering follower state: follower="Node at 192.168.15.111:8904 [Follower]" leader-address= leader-id=
[mux] 2024/01/09 08:35:57 cannot read header byte: tls: first record does not look like a TLS handshake
[mux] 2024/01/09 08:35:57 handler not registered for request from 192.168.15.111:63191: 71 (unsupported protocol?)
[cluster-join] 2024/01/09 08:35:57 failed to join via node at 192.168.15.111:8904: no leader
[cluster-join] 2024/01/09 08:35:57 failed to join cluster at [192.168.15.111:8904], after 1 attempt(s)
[store] 2024/01/09 08:35:57 reached expected bootstrap count of 1, starting cluster bootstrap
[store] 2024/01/09 08:35:57 cluster bootstrap failed: bootstrap only works on new clusters
[cluster-bootstrap] 2024/01/09 08:35:57 succeeded notifying all targets: [192.168.15.111:8904]
[cluster-join] 2024/01/09 08:35:59 failed to join via node at 192.168.15.111:8904: no leader
[cluster-join] 2024/01/09 08:35:59 failed to join cluster at [192.168.15.111:8904], after 1 attempt(s)
[cluster-bootstrap] 2024/01/09 08:35:59 succeeded notifying all targets: [192.168.15.111:8904]
2024-01-09T08:36:00.942-0300 [WARN] raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-01-09T08:36:00.942-0300 [INFO] raft: entering candidate state: node="Node at 192.168.15.111:8904 [Candidate]" term=18
2024-01-09T08:36:00.947-0300 [DEBUG] raft: voting for self: term=18 id=192.168.15.111
2024-01-09T08:36:00.953-0300 [DEBUG] raft: calculated votes needed: needed=1 term=18
2024-01-09T08:36:00.953-0300 [DEBUG] raft: vote granted: from=192.168.15.111 term=18 tally=1
2024-01-09T08:36:00.953-0300 [INFO] raft: election won: term=18 tally=1
2024-01-09T08:36:00.953-0300 [INFO] raft: entering leader state: leader="Node at 192.168.15.111:8904 [Leader]"
2024-01-09T08:36:00.953-0300 [INFO] raft: added peer, starting replication: peer=192.168.15.44
[store] 2024/01/09 08:36:00 first log applied since node start, log at index 227
2024-01-09T08:36:00.980-0300 [INFO] raft: pipelining replication: peer="{Nonvoter 192.168.15.44 192.168.15.44:8904}"
[store] 2024/01/09 08:36:00 20 confirmed committed log entries applied in 24.802ms, took 4.6805604s since open
[cluster-bootstrap] 2024/01/09 08:36:02 boot operation marked done
[rqlited] 2024/01/09 08:36:02 node HTTP API available at https://192.168.15.111:8906
[rqlited] 2024/01/09 08:36:02 connect using the command-line tool via 'rqlite -H 192.168.15.111 -p 8906'
[store] 2024/01/09 08:36:02 incremental snapshot created in 0s on node ID 192.168.15.111
2024-01-09T08:36:02.063-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:02 persisted snapshot 17-246-1704800162063 in 6.4689ms
2024-01-09T08:36:02.087-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:04 incremental snapshot created in 860.4µs on node ID 192.168.15.111
2024-01-09T08:36:04.103-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:04 persisted snapshot 17-246-1704800164103 in 0s
2024-01-09T08:36:04.110-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:06 incremental snapshot created in 600.8µs on node ID 192.168.15.111
2024-01-09T08:36:06.112-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:06 persisted snapshot 17-246-1704800166112 in 0s
2024-01-09T08:36:06.122-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:08 incremental snapshot created in 526.5µs on node ID 192.168.15.111
2024-01-09T08:36:08.141-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:08 persisted snapshot 17-246-1704800168141 in 0s
2024-01-09T08:36:08.150-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:10 incremental snapshot created in 600.3µs on node ID 192.168.15.111
2024-01-09T08:36:10.162-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:10 persisted snapshot 17-246-1704800170162 in 0s
2024-01-09T08:36:10.176-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:12 incremental snapshot created in 0s on node ID 192.168.15.111
2024-01-09T08:36:12.187-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:12 persisted snapshot 17-246-1704800172187 in 0s
2024-01-09T08:36:12.203-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:14 incremental snapshot created in 165.3µs on node ID 192.168.15.111
2024-01-09T08:36:14.214-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:14 persisted snapshot 17-246-1704800174214 in 0s
2024-01-09T08:36:14.224-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[store] 2024/01/09 08:36:16 incremental snapshot created in 0s on node ID 192.168.15.111
2024-01-09T08:36:16.236-0300 [INFO] raft: starting snapshot up to: index=246
[store] 2024/01/09 08:36:16 persisted snapshot 17-246-1704800176236 in 0s
2024-01-09T08:36:16.246-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
[rqlited] 2024/01/09 08:36:18 received signal "interrupt", shutting down
[http] 2024/01/09 08:36:18 closing HTTP service on 192.168.15.111:8906
[http] 2024/01/09 08:36:18 HTTP service on 192.168.15.111:8906 stopped: http: Server closed
[rqlited] 2024/01/09 08:36:18 stepping down as Leader before shutdown
2024-01-09T08:36:18.326-0300 [DEBUG] raft: starting leadership transfer: id=<nil> address=<nil>
2024-01-09T08:36:18.326-0300 [DEBUG] raft: cannot find peer
2024-01-09T08:36:18.326-0300 [INFO] raft: aborting pipeline replication: peer="{Nonvoter 192.168.15.44 192.168.15.44:8904}"
[store] 2024/01/09 08:36:18 store closed with node ID 192.168.15.111, listening on 192.168.15.111:8904
[rqlited] 2024/01/09 08:36:18 rqlite server stopped
From now on, everytime I try to start the cluster I’ll get this from the voter node:
failed to open store: failed to create snapshot store: check failed: journal mode to WAL: database disk image is malformed
What happens in the non-voter machine are multiple attempts to join cluster that naturally keep failing:
[cluster-join] 2024/01/09 08:37:35 failed to join via node at 192.168.15.111:8904: factory is not able to fill the pool: dial tcp 192.168.15.111:8904: connectex: No connection could be made because the target machine actively refused it.
[cluster-join] 2024/01/09 08:37:35 failed to join cluster at [192.168.15.111:8904], sleeping 3s before retry
[cluster-join] 2024/01/09 08:37:39 failed to join via node at 192.168.15.111:8904: factory is not able to fill the pool: dial tcp 192.168.15.111:8904: connectex: No connection could be made because the target machine actively refused it.
[cluster-join] 2024/01/09 08:37:39 failed to join cluster at [192.168.15.111:8904], sleeping 3s before retry
Context: - Select queries and /nodes queries are constantly being sent to each node when they are running properly. - Before interrumption: no write queries were being executed. - After the interrumption: as the cluster starts again, we check if the leader is ready and if it is ready we might perform /backup and then write queries so one of them could be in execution the first time the “raft: failed to take snapshot” error started happening in the log above. - We upgraded an existing v7 version cluster which already had been populated.
I can reproduce this issue on v8 variants (tested with v8.12.2 and more recent v8.15) but no on v7 variants.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 36 (24 by maintainers)
Commits related to this issue
- Update CHANGELOG.md https://github.com/rqlite/rqlite/issues/1581 — committed to rqlite/rqlite by otoolep 3 years ago
Great news, and thanks again for working through this issue – and providing the detailed repro steps. It was a real issue, and needed to be fixed.