nats-server: subscription count in subsz is wrong
SInce updating one of my brokers to 2.0.0 I noticed a slow increate in subscription counts - I also did a bunch of other updates like move to the newly renamed libraries etc so in order to find the cause I eventually concluded the server is just counting things wrongly.
Ignoring the annoying popup, you can see a steady increase in subscriptions.
Data below is from the below dependency embedded in another go process:
github.com/nats-io/nats-server/v2 v2.0.1-0.20190701212751-a171864ae7df
$ curl -s http://localhost:6165/varz|jq .subscriptions
29256
I then tried to verify this number, and assuming I have no bugs in the script below I think the varz counter is off by a lot, comparing snapshots of connz over time I see no growth reflected there not in connection counts nor subscriptions:
$ curl "http://localhost:6165/connz?limit=200000&subs=1"|./countsubs.rb
Connections: 3659
Subscriptions: 25477
I also captured connz output over time 15:17, 15:56 and 10:07 the next day:
$ cat connz-1562685506.json|./countsubs.rb
Connections: 3657
Subscriptions: 25463
$ cat connz-1562687791.json|./countsubs.rb
Connections: 3658
Subscriptions: 25463
$ cat connz-1562687791.json|./countsubs.rb
Connections: 3658
Subscriptions: 25463
Using the script here:
require "json"
subs = JSON.parse(STDIN.read)
puts "Connections: %d" % subs["connections"].length
count = 0
subs["connections"].each do |conn|
count += subs.length if subs = conn["subscriptions_list"]
end
puts "Subscriptions: %d" % count
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 87 (87 by maintainers)
Commits related to this issue
- Fix for #1065 (leaked subscribers from dq subs across routes) Signed-off-by: Derek Collison <derek@nats.io> — committed to nats-io/nats-server by derekcollison 5 years ago
- Fix for #1065 (leaked subscribers from dq subs across routes) Signed-off-by: Derek Collison <derek@nats.io> — committed to nats-io/nats-server by derekcollison 5 years ago
- Merge pull request #1079 from nats-io/leaksubs Fix for #1065 (leaked subscribers from dq subs across routes) — committed to nats-io/nats-server by derekcollison 5 years ago
I tracked down the issue and applied a brute force fix to make the tests pass. However this fix is not suitable for real code, I just hold a lock longer to synchronize some things to test my theory. It will cause a deadlock so need to come up with a better way. Will work on that tomorrow.
the 7290 would be go-nats fb0396ee0bdb8018b0fef30d6d1de798ce99cd05
ah damn, now I notice those are in fact go versions not client library versions…
An unclustered one does not appear to exhibit the same behaviour, regarding the memory though above I cannot 100% say the memory increase is NATS related since as I said it is an embedded instance and I have some other stuff going on. But being so closely related its worth considering.