nats-server: JetStream Cluster becomes inconsistent: catchup for stream stalled

Defect

Make sure that these boxes are checked before submitting your issue – thank you!

• Included nats-server -DV output
• Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve)

Versions of nats-server and affected client libraries used:

nats:2.6.2-alpine

OS/Container environment:

k8s

Context

• I don’t know how this problem arises, but it happens every few hours or days at max.
• The production system this is captured on has a message rate of about 1/s, with a few bursts every few hours of maybe 5/s (at least according to Grafana).
• The cluster consists of three nodes, and a stream with 3 replicas.
• After hours/days, one of the nodes start emitting Catchup for stream '$G > corejobs' stalled
• All consumers, where this node is the leader, stop getting messages.
• I dumped the jetstream data directory so you can get a look at the current state.

Steps or code to reproduce the issue:

• Start the cluster with the data and config provided here.
• Make sure, nats-cluster-2 is not the leader. (use nats stream cluster step-down if necessary)
• Send a message to the stream: nats pub core.jobs.notifications.SendPush foobar
• Observe the logs of nats-cluster-2 (which now throws warnings)

Additionally: check the message count in the stream; when nats-cluster-2 is the leader, it differs from when nats-cluster-0 or nats-cluster-1 are leaders.

Expected result:

Either:

1. All three nodes have the same data.
2. The node which can’t catch up is marked faulty and steps down from all leader positions

Also: JetStream should always be able to recover from inconsistent states (especially if there is still a majority of healthy nodes around).

Actual result:

• The cluster thinks it’s healthy.
• One node doesn’t receive any data anymore.

Additionally: depending on who becomes leader, the amount of messages varies (obviously, since it’s no longer synching). Node0: 310 messages, Node1: 310 messages, Node2: 81 messages