Umbraco-CMS: Fails to load NuCache

Which exact Umbraco version are you using? For example: 9.0.1 - don’t just write v9

9.4.1

Bug summary

Not sure if this is related to https://github.com/umbraco/Umbraco-CMS/pull/12209 but after upgrading from 9.2.0 to 9.4.1 it seems we get some errors loading NuCache.

We are importing some store nodes using Hangfire, which we tested on 9.2.0, but it seems after upgrading to 9.4.1 we get some errors as the Hangfire job fails with this exception.

System.AggregateException: Exceptions were thrown by listed actions. (An unexpected network error occurred. : 'C:\home\site\wwwroot\umbraco\Data\TEMP\NuCache\NuCache.Content.db')
 ---> System.IO.IOException: An unexpected network error occurred. : 'C:\home\site\wwwroot\umbraco\Data\TEMP\NuCache\NuCache.Content.db'
   at System.IO.FileStream.ReadNative(Span`1 buffer)
   at System.IO.FileStream.ReadSpan(Span`1 destination)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at CSharpTest.Net.IO.StreamCache.CachedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at CSharpTest.Net.IO.IOStream.ReadChunk(Stream io, Byte[] bytes, Int32 offset, Int32 length)
   at CSharpTest.Net.IO.TransactedCompoundFile.ReadBytes(Int64 position, Byte[] bytes, Int32 offset, Int32 length)
   at CSharpTest.Net.IO.TransactedCompoundFile.FileSection.Read(BlockRef& block, Boolean headerOnly, FGet fget)
   at CSharpTest.Net.IO.TransactedCompoundFile.Read(UInt32 handle)
   at CSharpTest.Net.Storage.BTreeFileStoreV2.TryGetNode[TNode](IStorageHandle handleIn, TNode& node, ISerializer`1 serializer)
   at CSharpTest.Net.Collections.BPlusTree`2.NodeCacheNone.Lock(NodePin parent, LockType ltype, NodeHandle child)
   at CSharpTest.Net.Collections.BPlusTree`2.RootLock..ctor(BPlusTree`2 tree, LockType type, Boolean exclusiveTreeAccess, String methodName)
   at CSharpTest.Net.Collections.BPlusTree`2.AddEntry[T](TKey key, T& info)
   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.Release(WriteLockInfo lockInfo, Boolean commit)
   at Umbraco.Cms.Infrastructure.PublishedCache.ContentStore.ScopedWriteLock.Release(Boolean completed)
   at Umbraco.Cms.Core.Scoping.ScopeContextualBase.<>c__1`1.<Get>b__1_1(Boolean completed, T item)
   at Umbraco.Cms.Core.Scoping.ScopeContext.EnlistedObject`1.Execute(Boolean completed)
   at Umbraco.Cms.Core.Scoping.ScopeContext.ScopeExit(Boolean completed)
   --- End of inner exception stack trace ---
   at Umbraco.Cms.Core.Scoping.ScopeContext.ScopeExit(Boolean completed)
   at Umbraco.Cms.Core.Scoping.Scope.<>c__DisplayClass106_0.<RobustExit>b__2()
   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Cms.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Cms.Core.Scoping.Scope.RobustExit(Boolean completed, Boolean onException)
   at Umbraco.Cms.Core.Scoping.Scope.DisposeLastScope()
   at Umbraco.Cms.Core.Scoping.Scope.Dispose()
   at Umbraco.Cms.Core.Services.Implement.ContentService.SaveAndPublish(IContent content, String culture, Int32 userId)

I have simplified the code a bit:

public interface IStoreSyncHandler
{
    [AutomaticRetry(Attempts = 0, OnAttemptsExceeded = AttemptsExceededAction.Fail)]
    [JobDisplayName("Sync stores")]
    public Task RunAsync(PerformContext context);
}

public class StoreSyncHandler : IStoreSyncHandler
{
    private readonly ILogger<StoreSyncHandler> _logger;

    private readonly IUmbracoContextFactory _umbracoContextFactory;
    private readonly IPublishedSnapshotAccessor _publishedSnapshotAccessor;
    private readonly IContentService _contentService;
    private readonly IIOHelper _ioHelper;
    private readonly MediaFileManager _mediaFileManager;
    private readonly GoogleGeocoder _geocoder;

    public StoreSyncHandler(
        ILogger<StoreSyncHandler> logger,
        IUmbracoContextFactory umbracoContextFactory,
        IPublishedSnapshotAccessor publishedSnapshotAccessor,
        IContentService contentService,
        IIOHelper ioHelper,
        MediaFileManager mediaFileManager,
        IOptions<GoogleMapsConfig> config)
    {
        _logger = logger;
        _umbracoContextFactory = umbracoContextFactory;
        _publishedSnapshotAccessor = publishedSnapshotAccessor;
        _contentService = contentService;
        _ioHelper = ioHelper;
        _mediaFileManager = mediaFileManager;

        _geocoder = new GoogleGeocoder(config.Value.ApiKey);
    }

    public async Task RunAsync(PerformContext context)
    {
        try
        {
            using (var cref = _umbracoContextFactory.EnsureUmbracoContext())
            {
                var cache = cref.UmbracoContext.Content;

                var root = cache.GetAtRoot().FirstOrDefault();
                if (root == null)
                    throw new Exception("No root was found.");

                var settings = root.FirstChild<ContentModels.Settings>();
                var storeArchive = root.FirstChild<ContentModels.StoreLocator>();

                if (storeArchive == null)
                    throw new Exception("No store archive was found.");

                var importFile = settings?.StoreImportFile;

                if (string.IsNullOrEmpty(importFile))
                {
                    await Task.CompletedTask;
                }

                if (!importFile.EndsWith(".csv", StringComparison.InvariantCultureIgnoreCase))
                    throw new Exception("Import file is not a CSV file.");

                var config = new CsvConfiguration(CultureInfo.InvariantCulture)
                {
                    Delimiter = ",",
                    DetectColumnCountChanges = true,
                    HasHeaderRecord = true,
                    HeaderValidated = null,
                    MissingFieldFound = null,
                    IgnoreBlankLines = false,
                    Mode = CsvMode.RFC4180,
                    // Ignore header case.
                    PrepareHeaderForMatch = args => args.Header.ToLower()
                };

                string path = _ioHelper.GetRelativePath(importFile);

                path = path.TrimStart(Constants.CharArrays.Tilde);

                context.WriteLine(ConsoleTextColor.Gray, $"CSV path: {path}");

                if (_mediaFileManager.FileSystem.FileExists(path))
                {
                    var stream = _mediaFileManager.FileSystem.OpenFile(path);
                    stream.Seek(0, SeekOrigin.Begin);

                    using (var reader = new StreamReader(stream))
                    using (var csv = new CsvReader(reader, config))
                    {
                        var records = new List<StoreRecord>();

                        try
                        {
                            while (csv.Read())
                            {
                                try
                                {
                                    var record = csv.GetRecord<StoreRecord>();
                                    records.Add(record);
                                }
                                catch (TypeConverterException ex)
                                {
                                    _logger.LogError(ex, "Failed reading record.");
                                }
                                catch (Exception ex)
                                {
                                    _logger.LogError(ex, "Failed reading record.");
                                }
                            }

                            // Unique records
                            records = records.Where(x => x.CustomerNumber.HasValue && x.CustomerNumber.Value > 0)
                                                .DistinctBy(x => x.CustomerNumber.Value)
                                                .ToList();

                            var customerNumbers = records.Select(x => x.CustomerNumber).ToArray();
                            var existingStores = new List<StoreRecord>();

                            const int pageSize = 500;
                            var page = 0;
                            var total = long.MaxValue;

                            while (page * pageSize < total)
                            {
                                var children = _contentService.GetPagedChildren(storeArchive.Id, page++, pageSize, out total);

                                foreach (var child in children)
                                {
                                    var customerNumber = child.GetValue<int?>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.CustomerNumber).Alias);

                                    if (customerNumber == null)
                                        continue;

                                    // Check if exists
                                    var found = records.FirstOrDefault(x => x.CustomerNumber == customerNumber);
                                    if (found == null)
                                        continue;

                                    existingStores.Add(found);

                                    await CreateOrUpdateNode(storeArchive.Key, child, found);
                                }
                            }

                            var newStores = records.Except(existingStores) ?? Enumerable.Empty<StoreRecord>();
                            var totalNewStores = newStores.Count();

                            foreach (var store in newStores)
                            {
                                await CreateOrUpdateNode(storeArchive.Key, null, store);
                            }
                        }
                        catch (BadDataException ex)
                        {
                            _logger.LogError(ex, "Failed reading CSV.");

                            throw;
                        }
                        catch (Exception ex)
                        {
                            _logger.LogError(ex, "Failed reading CSV.");

                            throw;
                        }
                    }
                }
            }
        }
        catch (Exception e)
        {
            var message = e.Message ?? string.Empty;
            
            _logger.LogError(e, "Exception: {0} | Message: {1} | Stack Trace: {2}", e.InnerException != null ? e.InnerException.ToString() : "", message, e.StackTrace);

            throw;
        }
    }

    private async Task CreateOrUpdateNode(Guid parentId, IContent node, StoreRecord store)
    {
        if (store == null)
            return;

        string statusValue = store.Status?.Trim();
        statusValue = statusValue?.Length >= 1 ? statusValue.Substring(0, 1) : null;

        GoogleAddress googleAddress = null;

        string title = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Title).Alias);
        string location = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Location).Alias);
        string address = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Address).Alias);
        string zip = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Zip).Alias);
        string city = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.City).Alias);
        string country = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Country).Alias);
        string email = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Email).Alias);
        string phone = node?.GetValue<string>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Phone).Alias);

        var status = node?.GetValue<PartnerStatus>(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Status).Alias);
        var partnerStatus = statusValue == null ? (PartnerStatus?)null : Enum.TryParse(statusValue, true, out PartnerStatus ps) ? ps : default;

        bool isNew = node == null;

        bool isDiffStatus = status != partnerStatus;

        int diffCount = 0;

        bool isDiffAddress = (Comparison.AreEqual(store.Address, address, ref diffCount) &&
                            Comparison.AreEqual(store.Zip, zip, ref diffCount) &&
                            Comparison.AreEqual(store.City, city, ref diffCount) &&
                            Comparison.AreEqual(store.CountryName, country, ref diffCount)) == false;

        bool isDiffProps = isDiffAddress || isDiffStatus || (Comparison.AreEqual(store.Name, title, ref diffCount) && Comparison.AreEqual(store.Email, email, ref diffCount) && Comparison.AreEqual(store.Phone, phone, ref diffCount)) == false;

        // If not new entry or values of properties hasn't changed we don't need to continue
        if (!isNew && !isDiffProps)
            return;

        // Lookup address if location not is set or address has changed
        if (string.IsNullOrEmpty(location) || isDiffAddress)
        {
            googleAddress = await GetAddress(store.Address, store.Zip, store.City, store.CountryName);
        }

        var name = $"{store.CustomerNumber} - {store.Name}";

        node ??= _contentService.Create(name, parentId, ContentModels.Store.ModelTypeAlias);

        node.Name = name;

        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.CustomerNumber).Alias, store.CustomerNumber);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Title).Alias, store.Name);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Address).Alias, store.Address);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Zip).Alias, store.Zip);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.City).Alias, store.City);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Country).Alias, store.CountryName);

        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Email).Alias, store.Email);
        node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Phone).Alias, store.Phone);

        if (partnerStatus != null)
        {
            node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Status).Alias, partnerStatus);
        }

        if (googleAddress != null)
        {
            var map = new Our.Umbraco.GMaps.Models.Map();

            map.Address.Coordinates = new Our.Umbraco.GMaps.Models.Location
            {
                Latitude = googleAddress.Coordinates.Latitude,
                Longitude = googleAddress.Coordinates.Longitude
            };

            map.Address.FullAddress = googleAddress.FormattedAddress;
            map.Address.PostalCode = store.Zip;
            map.Address.City = store.City;
            map.Address.State = "";
            map.Address.Country = store.CountryName;

            var mapJson = Newtonsoft.Json.JsonConvert.SerializeObject(map);

            // Store JSON in GMaps property editor
            node.SetValue(ContentModels.Store.GetModelPropertyType(_publishedSnapshotAccessor, x => x.Location).Alias, mapJson);
        }

        _contentService.SaveAndPublish(node);
    }

    private async Task<GoogleAddress> GetAddress(string address, string zip, string city, string country)
    {
        try
        {
            var results = await _geocoder.GeocodeAsync(address, city, "", zip, country);
            var result = results.FirstOrDefault(x => !x.IsPartialMatch);

            if (result == null || result.Coordinates == null)
                return null;

            return result;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, $"Failed to lookup address: {address}, {zip} {city}, {country}.");
        }

        return null;
    }
}

Furthermore when reloading memory cache from Published Status dashboard it fails.

Also when saving and publish a content node it fails.

The site is hosted on Azure Web Apps, but not sure if that makes a difference.

Specifics

No response

Steps to reproduce

Import some content nodes via Hangfire and notice the NuCache may be corrupt. It may require a larger amount of data to reproduce.

In this specific project they have 7347 store nodes.

Expected result / actual result

No response