Umbraco-CMS: The process cannot access the file NuCache.Content.db because it is being used by another process

PR: https://github.com/umbraco/Umbraco-CMS/pull/5924

Something is happening in Azure WebApps where the NuCache.Content.db file is locked causing the site to hang. I’ve attached the log file for reference.

The exception is as follows:

Umbraco.Core.Exceptions.BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details.

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file 'D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at CSharpTest.Net.IO.TransactedCompoundFile..ctor(Options options)
   at CSharpTest.Net.Storage.BTreeFileStoreV2..ctor(Options options)
   at CSharpTest.Net.Collections.BPlusTree`2.OptionsV2.CreateStorage()
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheBase..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNormal..ctor(BPlusTreeOptions`2 options)
   at CSharpTest.Net.Collections.BPlusTree`2..ctor(BPlusTreeOptions`2 ioptions)
   at Umbraco.Web.PublishedCache.NuCache.DataSource.BTree.GetTree(String filepath, Boolean exists)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService..ctor(Options options, IMainDom mainDom, IRuntimeState runtime, ServiceContext serviceContext, IPublishedContentTypeFactory publishedContentTypeFactory, IdkMap idkMap, IPublishedSnapshotAccessor publishedSnapshotAccessor, IVariationContextAccessor variationContextAccessor, IUmbracoContextAccessor umbracoContextAccessor, ILogger logger, IScopeProvider scopeProvider, IDocumentRepository documentRepository, IMediaRepository mediaRepository, IMemberRepository memberRepository, IDefaultCultureAccessor defaultCultureAccessor, IDataSource dataSource, IGlobalSettings globalSettings, ISiteDomainHelper siteDomainHelper, IEntityXmlSerializer entitySerializer, IPublishedModelFactory publishedModelFactory, UrlSegmentProviderCollection urlSegmentProviders)

...

This is on Umbraco 8.0.1 which was upgraded from Umbraco 8.0.0. (attachment will need to be changed from .txt to.json)

UmbracoTraceLog.RD2818786B7D96.20190319.txt

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 5
Comments: 182 (142 by maintainers)

Commits related to this issue

Add Lucence Directory factory key to web.config Updated, as per advice here https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-476477802 — committed to rossvernal/UmbracoDocs by rossvernal 5 years ago

Most upvoted comments

The same config principles apply to v8 as v7 for running on Azure. We defo need to get the docs updated (PRs are welcome!) The current azure docs are here https://our.umbraco.com/Documentation/Getting-Started/Setup/Server-Setup/azure-web-apps

for v8 you will need these in appSettings:

<add key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />
<add key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.SyncTempEnvDirectoryFactory, Examine" />
- Alternatively depending on your requirements there is also Examine.LuceneEngine.Directories.TempEnvDirectoryFactory

+10

Shazwazza on Mar 26, 2019

I’m using 8.4.1 and just started getting this issue last night, 02/24/20. Will try upgrading and IgnoreLocalDb = true

Umbraco.Core.Exceptions.BootFailedException: Boot failed.

-> System.IO.IOException: The process cannot access the file D:\home\site\wwwroot\App_Data\TEMP\NuCache\NuCache.Content.db because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

danielibanez on Feb 25, 2020

Good news - the latest changes I’ve made have resolved the issue. The issue i mentioned here https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-509966706 is the reason why the files get locked, it is a race condition when booting the nucache system when there’s multiple appdomains starting up at once. The other good news is that i can confirm the above work around does work so you can use that if you want to for now - but do please test it out and report back!

With this patch, I’ve tested multiple times:

Site swap = OK
Toggling App Insights = OK
Site swap where one slot has App Insights and not the other = OK

Have made a draft PR here https://github.com/umbraco/Umbraco-CMS/pull/5854

Turns out @zpqrtbnk mentioned this race condition in another document last week, I guess it’s good that 2 people discover the same thing separate research 😉

I cannot replicate the problem locally (naturally) no matter how many appdomains i can manage to startup at the same time so it’s odd that Azure is so affected by this.

Although this solves the file lock issue, it doesn’t explain why the Semaphore lock logging output doesn’t match up. I will investigate more of that tomorrow.

Shazwazza on Jul 11, 2019

Zpqrtbnk please let me know if any of us can help expedite this if you do not yet have the information to properly troubleshoot and isolate the issue. We have several v8 projects in the works scheduled to go live end of October. Do you have any concept of when this might be resolved or when you will be able to accurately describe what is occuring? We just passed 3months since the bug was opened. Not exactly sure what your release cycle looks like for v8, but again I would hope this makes it into 8.1?

brreisner on Jun 26, 2019

@brreisner - I had a similar situation in a fresh environment I set up this week, and I found a workaround: disabling app preload at the IIS level - yes, I know we are talking about Azure App Services.

The reasoning for trying this is that the conflict is somehow related to the early initialization of the app pool before the previous lock has been released. Then I found this attribute in IIS config <application … preloadEnabled=“true / false” …> which is by default true in Azure Web Apps.

Setting it to false stopped my NuCache lock conflict from happening on each new deploy / swap.

The thing is: because this is set at the IIS level (applicationHost.config file) you cannot edit it directly on an Azure Web App. But you can modify it by adding a transformation file, as explained here: https://blogs.msdn.microsoft.com/benjaminperkins/2018/01/09/how-to-disable-preloadenabled-on-an-azure-app-service/

NOTES:

If the lock conflict is happening in a multi-slot App Service, you need to repeat this procedure for each of the slots, because they are separate applications on IIS.
If you are using Slot Swap and you are concerned about the warm-up time, you can still use an <applicationInitialization> block in web.config. It does not trigger a NuCache lock conflict, at least in my case.

JoseMarcenaro on Jun 1, 2019

@stodolos We had this exact problem on two of our Umbraco 8 sites running in Azure Web App. However the issue seems to be fixed now (3 days without a single crash). Here is what we did:

Upgraded Umbraco to 8.0.2
Disabled the Application Insights on the web apps (both production and stage deployment slots)
Completely removed the Application Insights resource in the resource group
Removed all app settings that were automatically applied in the configuration tab on all our web apps (most of them regarding application insights)

Then it worked and no crashes during swaps. I think the 4th step is the most important. See if you have any left over configurations you don’t want.

HannesWinston on May 3, 2019