nats-server: Insufficient resources when creating JetStream on NFS with max-bytes and replica=3 set

Defect

Versions of nats-server and affected client libraries used:

• NATS v2.7.0

OS/Container environment:

• AWS Fargate with EFS storage

or

• CentOS 7 + NFS setup like in #2792

Steps or code to reproduce the issue:

Start NATS cluster with jetstream storage dir pointing to NFS mount:

nats-server -D -p 4222 --cluster_name test-cluster --name "nats1" -cluster "nats://localhost:4248" -routes "nats://localhost:4248,nats://localhost:5248,nats://localhost:6248" -js -sd "/mnt/nfs/nats1"
nats-server -D -p 5222 --cluster_name test-cluster --name "nats2" -cluster "nats://localhost:5248" -routes "nats://localhost:4248,nats://localhost:5248,nats://localhost:6248" -js -sd "/mnt/nfs/nats2"
nats-server -D -p 6222 --cluster_name test-cluster --name "nats3" -cluster "nats://localhost:6248" -routes "nats://localhost:4248,nats://localhost:5248,nats://localhost:6248" -js -sd "/mnt/nfs/nats3"

Add stream with max-bytes and replica=3 set:

# nats str add
? Stream Name DATA
? Subjects to consume data
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Message size limit 10241024
? Maximum message age limit -1
? Maximum individual message size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
? Replicas 3
nats: error: could not create Stream: insufficient resources (10023)

Expected result:

The stream gets created just like when running on local filesystem.

Actual result:

nats: error: could not create Stream: insufficient resources (10023)