sysrepo: sr_apply_changes() slowdown under heavy load

Hi!

When upgrading sysrepo I realized that under high CPU load and multiple subscribers the latest devel version can be twice as slow when calling sr_apply_changes() with push oper data.

I created an example application to reproduce the issue, but I had to use cpulimiter to simulate a greater load (cpulimiter -f -l 15 oper_push_test). I ran it 100x times with 15% CPU limit and on average sr_apply_changes() took ~2000ms on devel and ~1000ms on the older version I’m using.

I’ve attached the example and also the script and docker files I’ve used to measure.

What could be the reason of this slowdown? In a real scenario these differences are seconds per apply. As far as I can tell sr_changes_notify_store() takes up most of the time.

Older versions: libyang: d4e888ae3d8d5d0b3b5ef58c303aa821721e9ffb sysrepo: 8482344504c8776342ef18b75f70391edc793d9a

oper_push_test.zip

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 22 (8 by maintainers)

Commits related to this issue

Most upvoted comments

I checked it out, but there wasn’t really a change in performance in my case. I’ll keep investigating though, thanks!

I see. I have measured the instructions again of the example you provided and got 503 M (old) vs. 383 M (new), the difference being in the LYB parsing functions. So the optimization should have significant effect.

But it seems the concurrency issues are caused by something else. Unless you provide a non-docker example that I could freely debug and analyze, I am not able to help you, though. Seems like a non-trivial issue and I do not have the spare time required to get to the bottom of it.

Also, it may very easily be a result of some lock bugfix, when a potential dead lock was fixed or even a missing lock added, which could have somewhat degraded the performance. In such a case the solution may be to change your app design, to reduce the contention.

Thanks, the problem was nicely visible when comparing callgrind outputs of older and newer sysrepo versions, even with no threads. The issue was caused indirectly by the changed design and it needed some additional code to handle properly. Now it should be even more efficient than the old version, please test it.

That seems like a nice example for me to try and measure, can you please provide it?

Sure, here are the callgrind results. callgrind seems to have changed the timings, cpulimit didn’t really do much difference, and so futex waits are shorter, the with the perftrace results.

callgrind_result.zip

~100 subscribers are pretty normal for a larger module in my use-case, so I would say it’s similar. I upped the number as much as I could, so the performance degradation is more prevalent.

Thanks, but looking at it now I am not sure it includes any useful data. All I learned from it is that more time is spent waiting on locks. If you could generate callgrind reports, they should actually show what functions are causing the increased concurrency.

I will take a look at this issue but, suspecting it will require some effort, I am trying to solve simpler issues first, there are still some of those left. I appreciate the data presented here but @awesomenode, you could also measure the old sysrepo version using perf and post it here in the meantime, thanks.

Thanks for the reproducer, and just for the record, we’re also seeing slower operation with “today’s versions” when compared with stuff from the end of 2024. We haven’t done the profiling just yet because there’s a lot of opportunities for investigation in our system 😦.