Umbraco-CMS: New Preview engine in 7.15.1 causing SQL Timeouts that affect publishing

Since the upgrade to 7.15.1, we started experiencing a lots of SQL Timeouts during publishing. After a lot of digging, we found the root cause of the problem to be the new preview engine #5059

Umbraco version

I am seeing this issue on Umbraco version: 7.15.1 on prem (no cloud)

Reproduction

Bug summary

When you request a preview and at the same time you try to save, the save operation goes in timeout.

Specifics

Umbraco 7.15.1, 15k nodes with lots of grids, vorto, DTGE fields, nested content. cmsPreviewXml is around 8Gb

Steps to reproduce

With large database and empty preview cache, launch a preview. At the same time have another user save or publish a node.

Expected result

Save operation should work 😃 And preview should appear.

Actual result

You get a SQL Timeout error, and preview never appears. Reason is that the preview generation takes longer than the SQL command timeout. The same SQL statement locks the update of tables during the save operation, so also the save operation goes into time out. Here the detailed error from log file log-sample.txt

The reason we discovered after lots of digging is:

  • the preview sql statements takes 42 sec to execute
  • this sql select statement locks the cmsPreviewXml table
  • updates/inserts on the cmsPreviewXml table cannot happen until lock is released, which happens after the select statement for the preview is completed.

Here the SQL Lock reporting taken from SQL Trace: lock-report.txt

The workaround we implement is to set the preview to Legacy, and we are now experimenting with increasing the cache window, but I looks to me that this cache should never expire, as it takes 42sec to be recreated every time (this is the time it takes on the production DB, on a dev machine it takes over than 3 minutes to run). At least it should not lock other resources.

We have platinum support and we have already contacted support which recommended to keep on using legacy and upgrade to v8 (which is not feasible). It might work for us but maybe something to be improved in Umbraco code to cope with such situations.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 28 (28 by maintainers)

Most upvoted comments

great, will close this and tag for 7.15.5, there’s a few other things that need to go out in that release too.

Thank you. Since it took us 2 months to validate the fix, I guess it’s also our fault 😃 If we need to wait another month, it won’t be a big deal. If it goes further than June, then I guess it might become an issue. I’ll check with support (we have platinum support) how we can handle this.

I’m unsure ā€˜when’ exactly this will happen though so if you need an urgent fix I’d suggest either building from the source at this revision with a pre-release version and put the nuget package on a custom feed, or however you might hotfix something like that with your build system. There’s a couple issues still outstanding for this version so it won’t get shipped this sprint and will need to check internally if it can even be next sprint.

Hi @Shazwazza Code working better now… Been doing simultaneuous previews of different docs while publishing ~10 content items while preview is building… Currently no errors get reported, and I can see in log files docs are published and previews loaded correctly. I guess we have a winner now. How should we proceed? Do we get a new ā€œofficialā€ 7.15.5 release from HQ and use the updated nuget package? Our CD/CI doesn’t allow for a drop-in assembly. If we don’t get an official release soon, what would you suggest?

I’m looking into this now but what amazes me is that legacy preview ā€˜works’ for you because legacy preview will use an incredible amount of memory and with the size of your preview set it’s seems crazy you aren’t gettting out of memory exceptions.

If we page the query in BuildPreviewXmlCache that will help alleviate timeout locks. Have checked DB indexes and there doesn’t seem to be any missing/needed.

I’m also unsure as to why we keep all the versions in this table, it is unnecessary since we don’t use that value. We insert or update like this:

Database.InsertOrUpdate(previewPoco,
    //Since the table has a composite key, we need to specify an explit update statement
    "SET xml = @Xml, timestamp = @Timestamp WHERE nodeId=@NodeId AND versionId=@VersionId",
    new {NodeId = previewPoco.NodeId, VersionId = previewPoco.VersionId, Xml = previewPoco.Xml, Timestamp = previewPoco.Timestamp});

Where we could just as easily ignore the versionId check so that it just always updates the row with the matching node id. Update aha, it’s a legacy thing šŸ˜• There’s legacy code that does query and rely on this version unfortunately. That code curiously is used in some cases by the legacy preview engine. i.e. in document.ToPreviewXml

I would suggest that this could be attempted:

  • I will add paging to the query
  • You could have an event handler in ContentService.Saved that deletes all rows in the preview xml table that is not the current version/nodeid when using the non-legacy preview engine

I’ll post a build with the paging soon

Wow, 8GB in the previewXml table! 🤯

Good find and surprising that it still performs as well as it does I guess. I’ll have to send this off for investigation, not sure what can be done about it.