Umbraco-CMS: Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1
Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)
11.3.1
Bug summary
Examine index gets corrupt and can’t view or manage the Examine dashboard and any content trying to read index for display purpose becomes empty.
Happens on version 11.3.2, but also on 13.1.1 with the only solution to complete delete Examine folder and restart the application.
Issue is already discussed on Our.
Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\MembersIndex\segments_vd")))
at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)
at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Umbraco.Cms.Infrastructure.Examine.ConfigurationEnabledDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Examine.Lucene.Directories.DirectoryFactoryBase.<>c__DisplayClass2_0.<Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory>b__0(String s)
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Examine.Lucene.Directories.DirectoryFactoryBase.Examine.Lucene.Directories.IDirectoryFactory.CreateDirectory(LuceneIndex luceneIndex, Boolean forceUnlock)
at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass1_0.<.ctor>b__0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
at System.Lazy`1.CreateValue()
at Examine.Lucene.Providers.LuceneIndex.PerformIndexItemsInternal(IEnumerable`1 valueSets, CancellationToken cancellationToken)
at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass49_0.<PerformIndexItems>b__0()
at Examine.Lucene.Providers.LuceneIndex.<>c__DisplayClass73_0.<QueueTask>b__0(Task x)
Specifics
For an unknown reason the index gets corrupt and bricks the back office dashboard.
Application is hosted on Azure and config is applied as per this guide: https://docs.umbraco.com/umbraco-cms/v/10.latest-lts/fundamentals/setup/server-setup/azure-web-apps
Steps to reproduce
N/A
Expected result / actual result
Expected to be able to at least view the dashboard and rebuild indexes if they get corrupt.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 80 (35 by maintainers)
Examine 3.3.0 has been published, release notes are here https://github.com/Shazwazza/Examine/releases/tag/v3.3.0
We are running into this problem as well on a 13.1.0 installation running on a single Azure App Service. The error seems to start at random, a deletion of the examine folder and a site restart fixes it, but not for long.
We can’t even use the api to rebuild the index, as that will throw the same “invalid deletion count” error.
This should definitely have a higher priority in getting fixed 🙏
Good news everyone, Examine 3.7.0 has been released which should resolve this (again) https://github.com/Shazwazza/Examine/releases/tag/v3.7.0
@Migaroez As @paulsterling writes here: https://github.com/Shazwazza/Examine/issues/382#issuecomment-2189532338 this is also an issue on a default Umbraco Cloud v. 13.4.0 configuration. From your statement, that would mean, that the build-in configuration in Umbraco Cloud is faulty as well?
The updates for SyncedFileSystemDirectoryFactory can be found here https://github.com/Shazwazza/Examine/pull/387. Essentially, this will just allow the site to recover if the main or local index has become corrupt for whatever reason. It may mean that index rebuilds occur in these scenarios but at least the site will bootup. I’ve also added an options (not enabled by default) to run a repair on the index. This may result in documents being deleted, but in some cases, those documents may have been legitimate deletions before the index was committed. That would mean that index rebuilds don’t occur but potentially document loss in the index - which is why I’ll keep it optional. Hoping to get a release out this week.
Ohh my, this is such a mess 😬 I do look forward to a SomewhatFixedUmbracoSyncTempEnvFileSystemDirectoryFactory or whatever naming is gonna rival the best Microsoft can come up with 😄
@Shazwazza thank you for taking the time to help out on this 🙏 Even though this really is a responsibility of HQ, as I see it. They ship a product, that is clearly broken even when running on their own infrastructure.
Perhaps ExamineX should have a better bundling option on Cloud setups going forward 🤔
I agree… sorry, it seems I inadvertently closed it when referencing this issue in an related PR from another project.
Hey y’all, we are aware that these issues exist, but we have strong suspicions these things are popping up due to misconfiguration of azure web apps/load balancing/slot swapping. To be able to determine whether these issues are related to misconfiguration/bugs in examine/bugs in Umbraco, we are trying to build a troubleshooting guide, this will take some time still. In the meantime I advice you to read up on the issue @kevinstampe linked over on the examine repo (https://github.com/Shazwazza/Examine/issues/382#issuecomment-2158962439) and the docs concerning azure/examine https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/load-balancing/azure-web-apps#lucene-examine-configuration https://docs.umbraco.com/umbraco-cms/fundamentals/setup/server-setup/azure-web-apps#what-are-azure-web-apps
You can check a few of these configurations with @warrenbuckley’s RuntimeValidators https://github.com/Gibe/Umbraco.Community.RuntimeValidators
Hope to get back to you soon.
@Shazwazza I have upgraded to Examine 3.7.1 and don’t see this issue anymore.
I also created a PR to update Examine in Umbraco : https://github.com/umbraco/Umbraco-CMS/pull/19186
We installed version 3.7.0 and have had it on dev environment since March 20. Deployed to live environment on April 11 at 7 am, the site seemed fine during the day but it went down after business hours and the log is now again full of
Lucene.Net.Index.CorruptIndexException: invalid deletion count: 96 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=“C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\ExternalIndex\segments_2q4”)))
We also experience this issue in our load balanced environment (Umbraco 13 currently, but seen on 7,9,10,13), albeit infrequently. As per the load balanced docs (please see below) we have the scheduling publisher box (single instance) set as ‘SyncedTempFileSystemDirectoryFactory’ and subscriber box (load balanced instance) set as ‘TempFileSystemDirectoryFactory’
The issue seems to occur sometimes when our overnight data importers that consume external API data and then need to update Umbraco’s content interact with one of the scheduled publisher indexes, which seems to have got itself into an unrecoverable state. When this occurs I have to manually delete the indexes from the temp folder on the server and restart. Example log:
{"@t":"2025-03-09T02:25:04.9882276Z","@mt":"{IndexName} index is corrupt, a new one will be created","@l":"Error","@x":"Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(MMapIndexInput(path=\"C:\\home\\site\\wwwroot\\umbraco\\Data\\TEMP\\ExamineIndexes\\MembersIndex\\segments_28c\")))\r\n at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)\r\n at Lucene.Net.Index.IndexFileDeleter..ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)\r\n at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)","IndexName":"MembersIndex","SourceContext":"Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory","ProcessId":8276,"ProcessName":"w3wp","ThreadId":29,"ApplicationId":"ea6c9907b5dfb977abd279e5dd977d85f29f78cb","MachineName":"wn1xsdwk0008SF","Log4NetLevel":"ERROR"}I have seen this on multiple installs on multiple versions of Umbraco, so has it ever been considered that if this occurs then there should be final line of defence coding to physically delete the Examine cache from the file system and force it to rebuild? I know that there is code in place to try and create a new index, but when this issue occurs, the underlying cache seems to be beyond repair so requires deleting and rebuilding.
No, I use Visual Studio 2022 Publish to Azure to deploy the code and don’t use any Deployment slots and the app is the only application who points to the Database.
@PeterAtValantic
Then… we need to figure out how to replicate this in an isolated test setup environment - just like I did for this exact issue already, but clearly there’s some other strange way to continue to break the indexes.
See https://github.com/Shazwazza/Examine/blob/support/3.x/src/Examine.Test/Examine.Lucene/Directories/SyncedFileSystemDirectoryFactoryTests.cs
@markadrake all of your errors appear to be related to “Lucene.Net.Store.LockObtainFailedException” for the DeliveryApiContentIndex. This is a new one I haven’t seen but considering the location reported:
That location indicates that this index is not running on the local temp/fast drive and instead operating on the shared network drive. Please review your settings for “LuceneDirectoryFactory” for Azure hosting:
@Shazwazza I had this error last day as well. I am on Umbraco 13.5.3 and have updated all nuget packages and Examine package is on 3.6.0. I am hosting on Azure as well.
This is how my appsettings.json looks like:
"Umbraco": { "CMS": { "RuntimeMinification": { "UseInMemoryCache": true, }, "Global": { "MainDomLock": "FileSystemMainDomLock" }, "Hosting": { "LocalTempStorageLocation": "EnvironmentTemp" }, "Examine": { "LuceneDirectoryFactory": "SyncedTempFileSystemDirectoryFactory" } }This is the umbraco log file: { “@t”: “2025-02-03T11:55:04.0921069Z”, “@mt”: “{IndexName} index is corrupt, a new one will be created”, “@l”: “Error”, “@x”: “Lucene.Net.Index.CorruptIndexException: invalid deletion count: 2 vs docCount=1 (resource: BufferedChecksumIndexInput(SimpleFSIndexInput(path="C:\home\site\wwwroot\umbraco\Data\TEMP\ExamineIndexes\InternalIndex\segments_1g1")))\r\n at Lucene.Net.Index.SegmentInfos.Read(Directory directory, String segmentFileName)\r\n at Lucene.Net.Index.IndexFileDeleter…ctor(Directory directory, IndexDeletionPolicy policy, SegmentInfos segmentInfos, InfoStream infoStream, IndexWriter writer, Boolean initialIndexExists)\r\n at Lucene.Net.Index.IndexWriter…ctor(Directory d, IndexWriterConfig conf)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.GetIndexWriter(Directory mainDir, OpenMode openMode)\r\n at Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory.TryGetIndexWriter(OpenMode openMode, Directory luceneDirectory, Boolean createNewIfCorrupt, String indexName, IndexWriter& indexWriter)”, “IndexName”: “InternalIndex”, “SourceContext”: “Examine.Lucene.Directories.SyncedFileSystemDirectoryFactory”, “ProcessId”: 2532, “ProcessName”: “w3wp”, “ThreadId”: 12,
“Log4NetLevel”: “ERROR” }
Another stacktrace of today:
So, how to solve this?
@mbogunovic thanks a bunch for the feedback. Maybe @bergmania, @nul800sebastiaan, or others may resolve this ticket? If there are existing/future issues, will be best they are opened on their own.
@tommilleruk yes, see above. Upgrade to examine 3.4.0 and tell us how it goes.
Sure, i think if folks can update to 3.4.0 and see how it goes, than report back here. With any luck maybe we can close this in November.
@kows + @readingdancer + everyone - some good news. I managed to replicate another issues which is fixed. I can’t exactly replicate the one with the ‘invalid deletion count’ but I think it is directly related to the new fix.
I’ve published a beta here https://www.nuget.org/packages/Examine/3.3.1-beta.1
If anyone can please test, that would be great. This version will end up being 3.4.0 as it comes with performance improvements too and a small breaking change which reduces the default max results from 500 to 100 (if you need more, than use QueryOptions)
Not sure if this will help anyone, but on investigating our own instance of this issue, I stumbled across the fact that the Azure app service plan had been set up with Zone Redundancy, which requires a minimum of 3x instance count. This wasn’t made clear when provisioning the service - so the production site was running on multiple instances without proper load balancing configuration. I’m sure this contributed in some way to all the indexing issues we were having. I wonder if others are running on multiple workers without knowing. Checking the umbracoServer table for active instances would show this if so.
Hi @AndyButland This issue is not resolved yet. We are still experiencing this issue on Umbraco 13.4.1 and Examine 3.3.0 🤷
This is the error we see in the log:
Or is that because we use the SyncedTempFileSystemDirectoryFactory as opposed to the SyncedFileSystemDirectoryFactory mentioned in @Shazwazza 's post ?
Edit: I see now, going through the code, that setting the “LuceneDirectoryFactory” app setting to “SyncedTempFileSystemDirectoryFactory” does indeed actually end up using the SyncedFileSystemDirectoryFactory 👍
We’re still experiencing this issue having just updated to U13.5.1, our understanding was that Examine 3.3.0 was part of Umbraco 13.5. Site hostet in Azure, no load balancing, using the SyncedTempFileSystemDirectoryFactory. We had internal, external and a custom siteSearch index fail this morning. Quickfix of deleting indexes and restarting still works to fix it temporarily.