Umbraco-CMS: Performance: Degradation in throughput v13 vs 15
Which Umbraco version are you using?
15.3.0
Bug summary
While testing out v15 with a “real world database” from one of our projects I’m noticing quite a difference in throughput between v13 and v15.
I’ve taken the database from a v13 website and hooked it up with two clean Umbraco-projects, one for v13 and another for v15. In the Umbraco 15 version I configured the DocumentBreadthFirstSeedCount to 50000 to get all content into the cache.
I’m using the exact same views and code on both versions, it’s basically:
- Some traversing to create a menu based on the root node and it’s children.
- A container node with around 2500 descendants (in different depth of the tree).
The view that I’m testing with is a simple cshtml file that just loops over and render nodes with links.
To even get close to comparing the performance, I first had to apply this hack: https://github.com/umbraco/Umbraco-CMS/issues/18869
After this I ran a couple of simple load tests to compare the throughput.
Umbraco 13.7.2 Median load time: 9ms Request per second: 343
Umbraco 15.3.0 Median load time: 28ms Request per second: 131
I’m also noticing quite a different behavior when it comes to GC-pressure:
Umbraco 13: While the memory usage goes up, the GC-pressure is not that high.
Umbraco 15:
In Umbraco 15, the GC runs almost all the time.
I’m not sure if all these test are “state of the art” but I’m confident that there is a big difference in the “out of the box” performance between the versions and I really think that this is something that is important and that should be addressed in upcoming releases.
Specifics
No response
Steps to reproduce
See above
Expected result / actual result
No response
This item has been added to our backlog AB#51952
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 17 (13 by maintainers)
Hi, just wanted to add an update to our issue. We upgraded our solution in test to Umbraco 16 today and the issues regarding document load times (and to an extent cold boot times) seem to have been resolved by this. Average load times for a single document now averages on 50-60ms, so on our end things are now looking ok.
Cheers!
I spent some more time digging into this issue and I might have found something worth looking into.
The test case that I have is rendering a menu and a list of content, totally somewhere around 800 nodes are used during the request. I know that this is probably not the “typical use case” but it does bring some underlying issues to the surface.
As far as I understand, each time a content node is fetched from the new HybridCache, a
ContentCacheNodeis returned from the cache, passed to some factories that will create a newContentNodeand then call the ctor ofPublishedContentto create an new instance.Looking at some profiling, most of the time in my request is spent creating these
PublishedContentinstances.Since the
DocumentCacheServiceis used to fetch content when traversing the tree, each call toGetNodeAsync(Guid key, bool preview)will go trough the process of fetchingContentCacheNodefrom memory or L2-cache and then create an instance ofPublishedContent. The same thing happens when.Url()is called, even if the same node was already fetched and passed trough the factories this process would be repeated. Notice in the trace above that the ctor ofPublishedContentwas called over 7000 times during this request.I made a very hacky test using
AppCaches.RequestCacheto see what would happen if I would cache thePublishedContentfor the duration of the request (see example below). This reduced the number of newPublishedContent-instances to half.The load time for this particular page is still 250-500% slower than it used to be in Umbraco 13 (the exact same code). (With my other hack from https://github.com/umbraco/Umbraco-CMS/issues/18869 applied as well).
I’m thinking that it would be very beneficial to explore options to improve this, I don’t know the exact constrains around
PublishedContentand HybridCache but some ideas and thoughts:Ideally the IMemoryCache would hold instances of ´PublishedContent´ and HybridCache should just use
ContentCacheNodewhen serializing toIDistributedCache(don’t know if that is even possible). IfPublishedContentneeds access to request-bound information likeVariationContextsomething could fetch from the memory cache and update the cached instance with the request-bound information. This would avoid the allocation of new instances ofPublishedContentandContentNodeon each request.Also, something similar to my “request-cache”-hack below could be implemented to avoid re-creating the same
PublishedContent-instance in the context of the same request. I’m aware that not all calls to DocumentCacheService will be in the context of a HttpRequest (Background jobs etc) but I don’t know if there is any other caching mechanism that can be used for this.I would be more then happy to provide more information should it be needed.
Code from the hacked
DocumentCacheServiceI would certainly not consider this “resolved”. 15/16 is still significantly slower than 13 in almost all cases that we have come across. It seems there is a lot of work to do here, but we need better metrics/details on what is going on. I can say that the database is under significant load during “normal” site operation whereas it was not previously.
Just coming across this now as we’re seeing significant performance issues in V15. We’re seeing 50,000K SQL calls on first boot, which when combined with production stampeding means the site cannot boot.
Hi again @Migaroez!
I re-ran my tests on the latest code in
main(v16.1-preview), on the same machine, same SQL Server instance and same database, this is the results:Umbraco 13
RPS: 229 Median load time: 33ms Startup to first render:
Umbraco 16.1-preview
RPS: 50 Median load time: 146ms Startup to first render:
I did not have the time to recreate a new fresh db so there might still be some items in recycle bin and links to “dead” content in this db, but since sites are upgraded this scenario is far from unlikely.
There are quite some noticeable regressions here:
IgnoreLocalDb=false.I revisited this and looked closer
When I tried to confirm this on v13, the ctor of
PublishedContentis only hit during initial load, on subsequent page loads it’s not hit and the fact that theIPublishedContent-instances keeps being re-created in v15/v16, both during the same request and on subsequent requests is the main problem here.Looking at the ctor of e.g.
PublishedContentit takes in theContentNode-instance from the cache,IElementCache(singleton) andIVariationContextAccessor(singleton), it kind of feels like it would be possible to cache it in some way.I did try to just “slam in” a cache that caches the
IPublishedContent-instance inDocumentCacheService.GetNodeAsync()which made the test deliver 183 RPS, closer to v13 but not all the way.From what I understand, the new HybridCache promised faster startup times but in this test case it’s quite the opposite. I do get that there might be some 1% edge cases where a site with 100s of thousands of nodes loads faster, but I’m guessing that the vast majority of sites has less than 10k items, most sites does not have a Redis-cache (and should not need to either).
I think that these findings represents more than one issue and I also think that HybridCache might have limitations that makes these hard to get around.
In an ideal world it should be possible to:
Seed the in memory cache in a more efficient way. Currently the
SeedAsync()-implementation inDocumentCacheServicejust loops over the seed keys and fetches them from HybridCache on after the other, this is a lot less efficient than loading them all from the database in one go (or using paging as well).Cache the
IPublishedContent-instance in memory and only useContentCacheNodeto serialize if a 2nd level cache is used. This is not supported with HybridCache (as far as I know) but could maybe be something to contribute and/or push Microsoft towards OR it HybridCache could be abstracted to allow different modes.I can also see a alternative introduce an option where one could maybe opt-out of hybrid cache and just load everything into memory like in the “good old days” if they we want to. Personally I’ve always seen this as a good thing with Umbraco. Could maybe be different implementations of
IDocumentCacheServicedepending on the needs of the implementing website.I understand that this is probably a bigger commitment and not something that can be fixed with duct tape, but I do encourage you and the team to talk about this as there is a window for fixes with breaking changes now before V17.
As always, I’m more than happy to help out and let me know if you think I should be more concreate on any of the ideas.
All the best! 🚀