nats.go: Jetstream KV watcher does not watch after NATS server restart

KV watcher does not do its job after Nats server is restarted.

nats.go version v1.17.0 nats server v2.9.1

Steps or code to reproduce the issue:

Implement simple watch-all logic.
Run nats server and client. Make updates with keys in a bucket. See, updates are coming.
Restart nats server

watcher, err := flowsKV.WatchAll(nats.Context(ctx))
	if err != nil {
		return err
	}
	defer watcher.Stop()
	for {
		select {
		case update := <-watcher.Updates():
			if update == nil {
				break
			}
			
		case <-ctx.Done():
			return nil
		}
	}

Expected result:

After nats restart watcher backs to printing latest KV updates

Actual result:

Watcher gives no updates Error happens time to time nats: consumer not active on connection [101] for subscription on “$KV.bucket.>”

js.Subscribe() without nats.OrderedConsumer() option behaves correctly, but this option is built-in into kv.Watch…

Thanks.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 19 (17 by maintainers)

Most upvoted comments

Seems to me that during a single invocation of ordered consumer it would know it’s state - what msg it received last.

So if the consumer is lost it can recreate it to continue from last known position.

It would have to do this anyway regardless of storage type since many things can happen to consumers.

If it did that it would be reliable even with memory storage.

ripienaar on Sep 28, 2022

$ nats --context nats_development kv watch X '>'
[2022-09-27 10:55:45] PUT X > y: z
19:49:37 Disconnected due to: EOF, will attempt reconnect
19:49:41 Reconnected [nats://0.0.0.0:44855]
19:49:54 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:50:04 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:50:14 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:50:24 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:50:34 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:50:44 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:50:54 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active
19:51:04 Unexpected NATS error from server nats://0.0.0.0:44855: nats: consumer not active

Here I restarted the server mid-watch, the watcher never worked again and consumer was not recreated. Something not happy with ordered consumers recovery logic.

ripienaar on Sep 27, 2022