nats-server: subscription count in subsz is wrong

SInce updating one of my brokers to 2.0.0 I noticed a slow increate in subscription counts - I also did a bunch of other updates like move to the newly renamed libraries etc so in order to find the cause I eventually concluded the server is just counting things wrongly.

graph

Ignoring the annoying popup, you can see a steady increase in subscriptions.

Data below is from the below dependency embedded in another go process:

github.com/nats-io/nats-server/v2 v2.0.1-0.20190701212751-a171864ae7df
$ curl -s http://localhost:6165/varz|jq .subscriptions
29256

I then tried to verify this number, and assuming I have no bugs in the script below I think the varz counter is off by a lot, comparing snapshots of connz over time I see no growth reflected there not in connection counts nor subscriptions:

$ curl "http://localhost:6165/connz?limit=200000&subs=1"|./countsubs.rb
Connections: 3659
Subscriptions: 25477

I also captured connz output over time 15:17, 15:56 and 10:07 the next day:

$ cat connz-1562685506.json|./countsubs.rb
Connections: 3657
Subscriptions: 25463
$ cat connz-1562687791.json|./countsubs.rb
Connections: 3658
Subscriptions: 25463
$ cat connz-1562687791.json|./countsubs.rb
Connections: 3658
Subscriptions: 25463

Using the script here:

require "json"

subs = JSON.parse(STDIN.read)
puts "Connections: %d" % subs["connections"].length

count = 0

subs["connections"].each do |conn|
  count += subs.length if subs = conn["subscriptions_list"]
end

puts "Subscriptions: %d" % count

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 87 (87 by maintainers)

Commits related to this issue

Most upvoted comments

I tracked down the issue and applied a brute force fix to make the tests pass. However this fix is not suitable for real code, I just hold a lock longer to synchronize some things to test my theory. It will cause a deadlock so need to come up with a better way. Will work on that tomorrow.

the 7290 would be go-nats fb0396ee0bdb8018b0fef30d6d1de798ce99cd05

ah damn, now I notice those are in fact go versions not client library versions…

An unclustered one does not appear to exhibit the same behaviour, regarding the memory though above I cannot 100% say the memory increase is NATS related since as I said it is an embedded instance and I have some other stuff going on. But being so closely related its worth considering.