nats-server: Interest policy: nats server won't delete acked messages if consumers are not keeping up

Defect

when using Interest policy, nats server won’t delete acked messages if consumers are not keeping up

Make sure that these boxes are checked before submitting your issue – thank you!

• Included nats-server -DV output - unfortunately I have no access atm, please let me know if you need it
• Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve)

Versions of nats-server and affected client libraries used:

nats-server 2.7.2 python nats client 2.0.0

OS/Container environment:

docker container nats 2.7.2 alpine

Steps or code to reproduce the issue:

1. create jetstream with following config

Configuration:

             Subjects: messages.*
     Acknowledgements: true
            Retention: File - Interest
             Replicas: 1
       Discard Policy: New
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

1. create two polling consumers, one for messages.msg1 and another for messages.msg2

Configuration:

        Durable Name: MSG_XXX_CONSUMER
           Pull Mode: true
      Filter Subject: messages.message1
      Deliver Policy: All
          Ack Policy: Explicit
            Ack Wait: 30s
       Replay Policy: Instant
     Max Ack Pending: 20,000
   Max Waiting Pulls: 512

1. consumers are slower than inbound producer but ack messages normally within configured timeframe
2. messages get stacked up inside nats server filesystem, waiting for consumer to pick them up
3. stop producer

Expected result:

messages should be eventually all consumed, acked and removed from nats file system

Actual result:

messages are all consumed, acked, but stay on nats file system. when we delete durable consumers from server and reregister consumers without producing any messages on stream, messages eventually disappear from nats server