Umbraco-CMS: The process cannot access the file '\NuCache\NuCache.Content.db' because it is being used by another process. When using Kubernetes Pods

Which exact Umbraco version are you using? For example: 9.0.1 - don’t just write v9

8.18.3

Bug summary

Umbraco version: 8.18.3 Multilingual site containing about 10000 pages. Nucache DB size: 1.4GB

I’m in a bit of a pickle. We have a site that runs on Kubernetes, in a docker container. This is something we’ve done before and it has worked. The setup is as follows:

1 CMS Pod, 3 Front-end Umbraco Pods 3 Nuxt applications communicating with the Umbraco Front-end pods.

We have with lower data volumes been all right, but since loading in 10000 pages, containing a bunch of baked HTML content, we cannot start up our sites. Each pod takes well over 10 minutes to start, and then we run into a

The process cannot access the file ‘C:\inetpub\wwwroot\site\App_Data\TEMP\NuCache\NuCache.Content.db’ because it is being used by another process.

Specifics

The issue occurs locally too. My local configuration is the same as the deployed one, except for these two app settings, but since this is Kube and docker, I don’t thing they’re toooo relevant. Could be wrong.

We have another site running in Kubernetes, and that works fine.

The only difference between those sites is that this one contains more logic, and that we have an Entity framework instance in this one. It is the published content cache that takes a long time to build though. Here is a dot trace output when the app tries to start up in a smaller data set instance:

<add xdt:Transform="Insert" key="Umbraco.Examine.LuceneDirectoryFactory" value="Examine.LuceneEngine.Directories.TempEnvDirectoryFactory, Examine" />
    <add xdt:Transform="Insert" key="Umbraco.Core.LocalTempStorage" value="EnvironmentTemp" />

We don’t use examine on the front-end servers, so WE’ve also implemented this:

public class DisabledExamineComposer : IUserComposer
    {
        public void Compose(Composition composition)
        {
            if (MofConfigurationManager.Current.IsFrontEndServer)
            {
                // replace the default
                composition.RegisterUnique<IUmbracoIndexesCreator, InMemoryExamineIndexFactory>();

                // replace the default populators
                composition.Register<MemberIndexPopulator, DisabledMemberIndexPopulator>(Lifetime.Singleton);
                composition.Register<ContentIndexPopulator, DisabledContentIndexPopulator>(Lifetime.Singleton);
                composition.Register<PublishedContentIndexPopulator, DisabledPublishedContentIndexPopulator>(
                    Lifetime.Singleton);
                composition.Register<MediaIndexPopulator, DisabledMediaIndexPopulator>(Lifetime.Singleton);
            }
        }
    }

SO I’m at a bit of a loss. It doesn’t feel like 10000 pages, plus about 5000 more images should do this to the cache. What are we doing wrong, and what can we do to fix the startup time and reduce the lock times?

One other note to mention: On the CMS, when saving pages, we got all these errors while the other pods were starting up:

We have a number of composers in the system that are reading from the cache, and on save, some also validate and update the database in custom tables, though I doubt that entity framework has too much to do with the cache loading, or lock period timeout exceeded errors… To test, I have disabled these to ensure that they don’t interfere, but we get the same issue…

We really need to hand this site over soon, and while everything is looking great, we cannot have more than one instance of the application running, and it doesn’t look great that we get all these lock timeouts…

Steps to reproduce

See above. You would need to create a container from the site, configure:

An azure Kubernetes Cluster with no persistent storage, D2V3 Node instance with 14GB memory. Deploy one CMS instance, 3 front-end Instances. DB is Azure SQL Server currently on a premium plan & 250 DTU to rule that out Load in 10000 pages containing block List editors and raw HTML We are using Plugins: Nexu, SkyBrud Url Redirect Manager Umbraco Forms, NewsletterStudio

Azure Blob storage for images

Expected result / actual result

Cache rebuilds quickly and app starts up quickly.