Umbraco-CMS: New Preview engine in 7.15.1 causing SQL Timeouts that affect publishing
Since the upgrade to 7.15.1, we started experiencing a lots of SQL Timeouts during publishing. After a lot of digging, we found the root cause of the problem to be the new preview engine #5059
Umbraco version
I am seeing this issue on Umbraco version: 7.15.1 on prem (no cloud)
Reproduction
Bug summary
When you request a preview and at the same time you try to save, the save operation goes in timeout.
Specifics
Umbraco 7.15.1, 15k nodes with lots of grids, vorto, DTGE fields, nested content. cmsPreviewXml is around 8Gb
Steps to reproduce
With large database and empty preview cache, launch a preview. At the same time have another user save or publish a node.
Expected result
Save operation should work š And preview should appear.
Actual result
You get a SQL Timeout error, and preview never appears. Reason is that the preview generation takes longer than the SQL command timeout. The same SQL statement locks the update of tables during the save operation, so also the save operation goes into time out. Here the detailed error from log file log-sample.txt
The reason we discovered after lots of digging is:
- the preview sql statements takes 42 sec to execute
- this sql select statement locks the
cmsPreviewXmltable - updates/inserts on the
cmsPreviewXmltable cannot happen until lock is released, which happens after the select statement for the preview is completed.
Here the SQL Lock reporting taken from SQL Trace: lock-report.txt
The workaround we implement is to set the preview to Legacy, and we are now experimenting with increasing the cache window, but I looks to me that this cache should never expire, as it takes 42sec to be recreated every time (this is the time it takes on the production DB, on a dev machine it takes over than 3 minutes to run). At least it should not lock other resources.
We have platinum support and we have already contacted support which recommended to keep on using legacy and upgrade to v8 (which is not feasible). It might work for us but maybe something to be improved in Umbraco code to cope with such situations.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 28 (28 by maintainers)
great, will close this and tag for 7.15.5, thereās a few other things that need to go out in that release too.
Thank you. Since it took us 2 months to validate the fix, I guess itās also our fault š If we need to wait another month, it wonāt be a big deal. If it goes further than June, then I guess it might become an issue. Iāll check with support (we have platinum support) how we can handle this.
Iām unsure āwhenā exactly this will happen though so if you need an urgent fix Iād suggest either building from the source at this revision with a pre-release version and put the nuget package on a custom feed, or however you might hotfix something like that with your build system. Thereās a couple issues still outstanding for this version so it wonāt get shipped this sprint and will need to check internally if it can even be next sprint.
Hi @Shazwazza Code working better now⦠Been doing simultaneuous previews of different docs while publishing ~10 content items while preview is building⦠Currently no errors get reported, and I can see in log files docs are published and previews loaded correctly. I guess we have a winner now. How should we proceed? Do we get a new āofficialā 7.15.5 release from HQ and use the updated nuget package? Our CD/CI doesnāt allow for a drop-in assembly. If we donāt get an official release soon, what would you suggest?
Iām looking into this now but what amazes me is that legacy preview āworksā for you because legacy preview will use an incredible amount of memory and with the size of your preview set itās seems crazy you arenāt gettting out of memory exceptions.
If we page the query in
BuildPreviewXmlCachethat will help alleviate timeout locks. Have checked DB indexes and there doesnāt seem to be any missing/needed.Iām also unsure as to why we keep all the versions in this table, it is unnecessary since we donāt use that value. We insert or update like this:
Where we could just as easily ignore the versionId check so that it just always updates the row with the matching node id. Update aha, itās a legacy thing š Thereās legacy code that does query and rely on this version unfortunately. That code curiously is used in some cases by the legacy preview engine. i.e. in
document.ToPreviewXmlI would suggest that this could be attempted:
Iāll post a build with the paging soon
Wow, 8GB in the previewXml table! š¤Æ
Good find and surprising that it still performs as well as it does I guess. Iāll have to send this off for investigation, not sure what can be done about it.