rqlite: Getting database disk image is malformed after rqlite interruption

What version are you running? v8.15.2

Are you using Docker or Kubernetes to run your system? No

Are you running a single node or a cluster? Cluster

Issue Description I’m facing an issue where rqlite might fall into “failed to open store: failed to create snapshot store: check failed: journal mode to WAL: database disk image is malformed” after being previously interrupted (killed).

Cluster information (2-nodes cluster): - Node ID: 192.168.15.111, Suffrage=Voter - Node ID: 192.168.15.44, Suffrage=NonVoter - Node Interrumpted: 192.168.15.111, Suffrage=Voter

Launch commands: 192.168.15.111(Voter): rqlited.exe -raft-log-level=DEBUG -node-id 192.168.15.111 -http-addr 192.168.15.111:8906 -raft-addr 192.168.15.111:8904 -join 192.168.15.111:8904 -bootstrap-expect 1 -raft-non-voter=false -node-ca-cert DatabaseCertificateAuthority.cer -node-cert DatabasePublicCertificate.cer -node-verify-client -node-key DatabasePrivateCertificate.pem -http-ca-cert DatabaseCertificateAuthority.cer -http-cert DatabasePublicCertificate.cer -http-verify-client -http-key DatabasePrivateCertificate.pem Local 192.168.15.44 (NonVoter): rqlited.exe -raft-log-level=DEBUG -node-id 192.168.15.44 -http-addr 192.168.15.44:8906 -raft-addr 192.168.15.44:8904 -join 192.168.15.111:8904 -raft-non-voter=true -node-ca-cert DatabaseCertificateAuthority.cer -node-cert DatabasePublicCertificate.cer -node-verify-client -node-key DatabasePrivateCertificate.pem -http-ca-cert DatabaseCertificateAuthority.cer -http-cert DatabasePublicCertificate.cer -http-verify-client -http-key DatabasePrivateCertificate.pem Local

After interrumption, if I start the cluster that’s what happens in the voter machine:

             _ _ _
            | (_) |
   _ __ __ _| |_| |_ ___
  | '__/ _  | | | __/ _ \   The lightweight, distributed
  | | | (_| | | | ||  __/   relational database.
  |_|  \__, |_|_|\__\___|
          | |               www.rqlite.io
          |_|
 
 [rqlited] 2024/01/09 08:35:56 rqlited starting, version 8, SQLite 3.44.0, commit unknown, branch unknown, compiler gc
 [rqlited] 2024/01/09 08:35:56 go1.20.2, target architecture is amd64, operating system target is windows
 [rqlited] 2024/01/09 08:35:56 launch command: rqlited.exe -raft-log-level=DEBUG -node-id 192.168.15.111 -http-addr 192.168.15.111:8906 -raft-addr 192.168.15.111:8904 -join 192.168.15.111:8904 -bootstrap-expect 1 -raft-non-voter=false -node-ca-cert DatabaseCertificateAuthority.cer -node-cert DatabasePublicCertificate.cer -node-verify-client -node-key DatabasePrivateCertificate.pem -http-ca-cert DatabaseCertificateAuthority.cer -http-cert DatabasePublicCertificate.cer -http-verify-client -http-key DatabasePrivateCertificate.pem  Local
 [rqlited] 2024/01/09 08:35:56 enabling node-to-node encryption with cert: DatabasePublicCertificate.cer, key: DatabasePrivateCertificate.pem, CA cert DatabaseCertificateAuthority.cer, mutual TLS enabled
 [rqlited] 2024/01/09 08:35:56 Raft TCP mux Listener registered with byte header 1
 [mux] 2024/01/09 08:35:56 TLS mux serving on 192.168.15.111:8904, advertising 192.168.15.111:8904
 [rqlited] 2024/01/09 08:35:56 preexisting node state detected in Local
 [cluster] 2024/01/09 08:35:56 service listening on 192.168.15.111:8904
 [rqlited] 2024/01/09 08:35:56 cluster TCP mux Listener registered with byte header 2
 [http] 2024/01/09 08:35:56 secure HTTPS server enabled with cert DatabasePublicCertificate.cer, key DatabasePrivateCertificate.pem, CA cert DatabaseCertificateAuthority.cer, mutual TLS enabled
 [http] 2024/01/09 08:35:56 execute queue processing started with capacity 1024, batch size 128, timeout 50ms
 [http] 2024/01/09 08:35:56 service listening on 192.168.15.111:8906
 [rqlited] 2024/01/09 08:35:56 HTTP server started
 [store] 2024/01/09 08:35:56 opening store with node ID 192.168.15.111, listening on 192.168.15.111:8904
 [store] 2024/01/09 08:35:56 ensuring data directory exists at Local
 [store] 2024/01/09 08:35:56 old snapshot directory Local\snapshots does not exist, nothing to upgrade
 [snapshot-store] 2024/01/09 08:35:56 store initialized using Local\rsnapshots
 [snapshot-store] 2024/01/09 08:35:56 checking consistency of snapshot store at Local\rsnapshots
 [snapshot-store] 2024/01/09 08:35:56 check complete
 [store] 2024/01/09 08:35:56 1 preexisting snapshots present
 [store] 2024/01/09 08:35:56 first log index: 1, last log index: 246, last applied index: 246, last command log index: 246:
 [store] 2024/01/09 08:35:56 created on-disk database at open
 2024-01-09T08:35:56.318-0300 [INFO]  raft: starting restore from snapshot: id=15-224-1704799965251 last-index=224 last-term=15 size-in-bytes=134144
 [store] 2024/01/09 08:35:56 initiating node restore on node ID 192.168.15.111
 [store] 2024/01/09 08:35:56 successfully opened database at Local\db.sqlite due to restore
 [store] 2024/01/09 08:35:56 node restored in 7.4983ms
 2024-01-09T08:35:56.326-0300 [INFO]  raft: snapshot restore progress: id=15-224-1704799965251 last-index=224 last-term=15 size-in-bytes=134144 read-bytes=134144 percent-complete="100.00%"
 2024-01-09T08:35:56.326-0300 [INFO]  raft: restored from snapshot: id=15-224-1704799965251 last-index=224 last-term=15 size-in-bytes=134144
 2024-01-09T08:35:56.326-0300 [INFO]  raft: initial configuration: index=3 servers="[{Suffrage:Voter ID:192.168.15.111 Address:192.168.15.111:8904} {Suffrage:Nonvoter ID:192.168.15.44 Address:192.168.15.44:8904}]"
 [rqlited] 2024/01/09 08:35:57 checking that any supplied join addresses don't serve HTTP(S)
 2024-01-09T08:35:57.220-0300 [INFO]  raft: entering follower state: follower="Node at 192.168.15.111:8904 [Follower]" leader-address= leader-id=
 [mux] 2024/01/09 08:35:57 cannot read header byte: tls: first record does not look like a TLS handshake
 [mux] 2024/01/09 08:35:57 handler not registered for request from 192.168.15.111:63191: 71 (unsupported protocol?)
 [cluster-join] 2024/01/09 08:35:57 failed to join via node at 192.168.15.111:8904: no leader
 [cluster-join] 2024/01/09 08:35:57 failed to join cluster at [192.168.15.111:8904], after 1 attempt(s)
 [store] 2024/01/09 08:35:57 reached expected bootstrap count of 1, starting cluster bootstrap
 [store] 2024/01/09 08:35:57 cluster bootstrap failed: bootstrap only works on new clusters
 [cluster-bootstrap] 2024/01/09 08:35:57 succeeded notifying all targets: [192.168.15.111:8904]
 [cluster-join] 2024/01/09 08:35:59 failed to join via node at 192.168.15.111:8904: no leader
 [cluster-join] 2024/01/09 08:35:59 failed to join cluster at [192.168.15.111:8904], after 1 attempt(s)
 [cluster-bootstrap] 2024/01/09 08:35:59 succeeded notifying all targets: [192.168.15.111:8904]
 2024-01-09T08:36:00.942-0300 [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
 2024-01-09T08:36:00.942-0300 [INFO]  raft: entering candidate state: node="Node at 192.168.15.111:8904 [Candidate]" term=18
 2024-01-09T08:36:00.947-0300 [DEBUG] raft: voting for self: term=18 id=192.168.15.111
 2024-01-09T08:36:00.953-0300 [DEBUG] raft: calculated votes needed: needed=1 term=18
 2024-01-09T08:36:00.953-0300 [DEBUG] raft: vote granted: from=192.168.15.111 term=18 tally=1
 2024-01-09T08:36:00.953-0300 [INFO]  raft: election won: term=18 tally=1
 2024-01-09T08:36:00.953-0300 [INFO]  raft: entering leader state: leader="Node at 192.168.15.111:8904 [Leader]"
 2024-01-09T08:36:00.953-0300 [INFO]  raft: added peer, starting replication: peer=192.168.15.44
 [store] 2024/01/09 08:36:00 first log applied since node start, log at index 227
 2024-01-09T08:36:00.980-0300 [INFO]  raft: pipelining replication: peer="{Nonvoter 192.168.15.44 192.168.15.44:8904}"
 [store] 2024/01/09 08:36:00 20 confirmed committed log entries applied in 24.802ms, took 4.6805604s since open
 [cluster-bootstrap] 2024/01/09 08:36:02 boot operation marked done
 [rqlited] 2024/01/09 08:36:02 node HTTP API available at https://192.168.15.111:8906
 [rqlited] 2024/01/09 08:36:02 connect using the command-line tool via 'rqlite -H 192.168.15.111 -p 8906'
 [store] 2024/01/09 08:36:02 incremental snapshot created in 0s on node ID 192.168.15.111
 2024-01-09T08:36:02.063-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:02 persisted snapshot 17-246-1704800162063 in 6.4689ms
 2024-01-09T08:36:02.087-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:04 incremental snapshot created in 860.4µs on node ID 192.168.15.111
 2024-01-09T08:36:04.103-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:04 persisted snapshot 17-246-1704800164103 in 0s
 2024-01-09T08:36:04.110-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:06 incremental snapshot created in 600.8µs on node ID 192.168.15.111
 2024-01-09T08:36:06.112-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:06 persisted snapshot 17-246-1704800166112 in 0s
 2024-01-09T08:36:06.122-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:08 incremental snapshot created in 526.5µs on node ID 192.168.15.111
 2024-01-09T08:36:08.141-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:08 persisted snapshot 17-246-1704800168141 in 0s
 2024-01-09T08:36:08.150-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:10 incremental snapshot created in 600.3µs on node ID 192.168.15.111
 2024-01-09T08:36:10.162-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:10 persisted snapshot 17-246-1704800170162 in 0s
 2024-01-09T08:36:10.176-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:12 incremental snapshot created in 0s on node ID 192.168.15.111
 2024-01-09T08:36:12.187-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:12 persisted snapshot 17-246-1704800172187 in 0s
 2024-01-09T08:36:12.203-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:14 incremental snapshot created in 165.3µs on node ID 192.168.15.111
 2024-01-09T08:36:14.214-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:14 persisted snapshot 17-246-1704800174214 in 0s
 2024-01-09T08:36:14.224-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [store] 2024/01/09 08:36:16 incremental snapshot created in 0s on node ID 192.168.15.111
 2024-01-09T08:36:16.236-0300 [INFO]  raft: starting snapshot up to: index=246
 [store] 2024/01/09 08:36:16 persisted snapshot 17-246-1704800176236 in 0s
 2024-01-09T08:36:16.246-0300 [ERROR] raft: failed to take snapshot: error="failed to close snapshot: journal mode to WAL: database disk image is malformed"
 [rqlited] 2024/01/09 08:36:18 received signal "interrupt", shutting down
 [http] 2024/01/09 08:36:18 closing HTTP service on 192.168.15.111:8906
 [http] 2024/01/09 08:36:18 HTTP service on 192.168.15.111:8906 stopped: http: Server closed
 [rqlited] 2024/01/09 08:36:18 stepping down as Leader before shutdown
 2024-01-09T08:36:18.326-0300 [DEBUG] raft: starting leadership transfer: id=<nil> address=<nil>
 2024-01-09T08:36:18.326-0300 [DEBUG] raft: cannot find peer
 2024-01-09T08:36:18.326-0300 [INFO]  raft: aborting pipeline replication: peer="{Nonvoter 192.168.15.44 192.168.15.44:8904}"
 [store] 2024/01/09 08:36:18 store closed with node ID 192.168.15.111, listening on 192.168.15.111:8904
 [rqlited] 2024/01/09 08:36:18 rqlite server stopped

From now on, everytime I try to start the cluster I’ll get this from the voter node: failed to open store: failed to create snapshot store: check failed: journal mode to WAL: database disk image is malformed

What happens in the non-voter machine are multiple attempts to join cluster that naturally keep failing:

	 [cluster-join] 2024/01/09 08:37:35 failed to join via node at 192.168.15.111:8904: factory is not able to fill the pool: dial tcp 192.168.15.111:8904: connectex: No connection could be made because the target machine actively refused it.
	 [cluster-join] 2024/01/09 08:37:35 failed to join cluster at [192.168.15.111:8904], sleeping 3s before retry
	 [cluster-join] 2024/01/09 08:37:39 failed to join via node at 192.168.15.111:8904: factory is not able to fill the pool: dial tcp 192.168.15.111:8904: connectex: No connection could be made because the target machine actively refused it.
	 [cluster-join] 2024/01/09 08:37:39 failed to join cluster at [192.168.15.111:8904], sleeping 3s before retry

Context: - Select queries and /nodes queries are constantly being sent to each node when they are running properly. - Before interrumption: no write queries were being executed. - After the interrumption: as the cluster starts again, we check if the leader is ready and if it is ready we might perform /backup and then write queries so one of them could be in execution the first time the “raft: failed to take snapshot” error started happening in the log above. - We upgraded an existing v7 version cluster which already had been populated.

I can reproduce this issue on v8 variants (tested with v8.12.2 and more recent v8.15) but no on v7 variants.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 36 (24 by maintainers)

Commits related to this issue

Most upvoted comments

Great news, and thanks again for working through this issue – and providing the detailed repro steps. It was a real issue, and needed to be fixed.