Umbraco-CMS: Examine index issues in Umbraco 8.6.0 Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed

We’ve recently upgraded to Umbraco 8.6.0 since we’ve experienced Examine index locking issues, described here: https://github.com/umbraco/Umbraco-CMS/issues/5035 The recommendation from Shazwazza was to upgrade to 8.6.0, see his comment: https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-599546750

We’ve upgraded, followed the instructions from https://our.umbraco.com/documentation/getting-started/setup/server-setup/azure-web-apps

However, once a few hours we’re getting Examine lock-related errors (details above). When we touch the web.config everything runs ok for a few hours then the problems reappear.

Umbraco version

I am seeing this issue on Umbraco version: 8.6.0, Examine v1.0.3, Lucene.Net v3.0.3

Reproduction

If you’re filing a bug, please describe how to reproduce it. Include as much relevant information as possible, such as:

Bug summary

We’re running in Azure, we’ve recently upgraded from 7 to 8.5.3 then to v8.6.0 With version 7 everything was working ok but we wanted to benefit from the latest version inprovments. With version 8.5.3, the website was working pretty well but we encountered the issues described here: https://github.com/Shazwazza/Examine/issues/161 so we upgraded.

Now all seems to work fine except some Examine-related exceptions described below:

When SAVING a document, the following exceptions appear in logs (think that they are related):

{"@t":"2020-04-23T12:45:43.8069949Z","@mt":"App is shutting down so index batch operation is ignored","@l":"Error","SourceContext":"Umbraco.Examine.UmbracoContentIndex","ProcessId":6220,"ProcessName":"w3wp","ThreadId":242,"AppDomainId":67,"AppDomainAppId":"LMW3SVC20377917ROOT","MachineName":"RD00155D0A79A5","Log4NetLevel":"ERROR"}
{"@t":"2020-04-23T12:45:43.8069949Z","@mt":"Exception","@l":"Error","@x":"System.ObjectDisposedException: The CancellationTokenSource has been disposed.\r\n   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()\r\n   at Examine.LuceneEngine.Providers.LuceneIndex.SafelyProcessQueueItems(Action`1 onComplete) in C:\\projects\\examine-qvx04\\src\\Examine\\LuceneEngine\\Providers\\LuceneIndex.cs:line 783\r\n   at Examine.LuceneEngine.Providers.LuceneIndex.PerformIndexItems(IEnumerable`1 values, Action`1 onComplete) in C:\\projects\\examine-qvx04\\src\\Examine\\LuceneEngine\\Providers\\LuceneIndex.cs:line 302\r\n   at Umbraco.Examine.UmbracoContentIndex.PerformIndexItems(IEnumerable`1 values, Action`1 onComplete) in D:\\a\\1\\s\\src\\Umbraco.Examine\\UmbracoContentIndex.cs:line 102\r\n   at Examine.Providers.BaseIndexProvider.IndexItems(IEnumerable`1 values) in C:\\projects\\examine-qvx04\\src\\Examine\\Providers\\BaseIndexProvider.cs:line 76\r\n   at Our.Umbraco.FullTextSearch.Components.PerformCacheTasks.PerformRun()","SourceContext":"Our.Umbraco.FullTextSearch.Components.PerformCacheTasks","ProcessId":6220,"ProcessName":"w3wp","ThreadId":242,"AppDomainId":67,"AppDomainAppId":"LMW3SVC20377917ROOT","MachineName":"RD00155D0A79A5","Log4NetLevel":"ERROR"}

When SEARCHING using top-right search button, the following exception is shown:

Unhandled controller exception occurred for request '{RequestUrl}'
Error: Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed
   at Lucene.Net.Index.IndexReader.EnsureOpen() in d:\\Lucene.Net\\FullRepo\\trunk\\src\\core\\Index\\IndexReader.cs:line 204
   at Lucene.Net.Index.DirectoryReader.GetFieldNames(FieldOption fieldNames) in d:\\Lucene.Net\\FullRepo\\trunk\\src\\core\\Index\\DirectoryReader.cs:line 1055
   at Examine.LuceneEngine.Providers.LuceneSearcher.GetAllIndexedFields() in C:\\projects\\examine-qvx04\\src\\Examine\\LuceneEngine\\Providers\\LuceneSearcher.cs:line 101
   at Examine.LuceneEngine.Providers.BaseLuceneSearcher.CreateQuery(String category, BooleanOperation defaultOperation, Analyzer luceneAnalyzer, LuceneSearchOptions searchOptions) in C:\\projects\\examine-qvx04\\src\\Examine\\LuceneEngine\\Providers\\BaseLuceneSearcher.cs:line 64
   at Examine.LuceneEngine.Providers.BaseLuceneSearcher.CreateQuery(String category, BooleanOperation defaultOperation) in C:\\projects\\examine-qvx04\\src\\Examine\\LuceneEngine\\Providers\\BaseLuceneSearcher.cs:line 49
   at Umbraco.Web.Search.UmbracoTreeSearcher.ExamineSearch(String query, UmbracoEntityTypes entityType, Int32 pageSize, Int64 pageIndex, Int64& totalFound, String searchFrom, Boolean ignoreUserStartNodes) in D:\\a\\1\\s\\src\\Umbraco.Web\\Search\\UmbracoTreeSearcher.cs:line 128
   at Umbraco.Web.Trees.ContentTreeController.Search(String query, Int32 pageSize, Int64 pageIndex, Int64& totalFound, String searchFrom) in D:\\a\\1\\s\\src\\Umbraco.Web\\Trees\\ContentTreeController.cs:line 331
   at Umbraco.Web.Editors.EntityController.SearchAll(String query) in D:\\a\\1\\s\\src\\Umbraco.Web\\Editors\\EntityController.cs:line 152
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__6.MoveNext()
   --- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
RequestUrl":"https://beta.magicbreaks.co.uk/umbraco/backoffice/UmbracoApi/Entity/SearchAll?query=te"

When OPEN Settings -> Examine management:

Error: The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.
Exception Details
System.InvalidOperationException: The 'ObjectContent`1' type failed to serialize the response body for content type 'application/json; charset=utf-8'.
Inner Exception
Lucene.Net.Store.AlreadyClosedException: this IndexWriter is closed
at Lucene.Net.Index.IndexWriter.EnsureOpen(Boolean includePendingClose) in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 852
   at Lucene.Net.Index.IndexWriter.get_Directory() in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexWriter.cs:line 2164
   at Umbraco.Examine.LuceneIndexDiagnostics.get_Metadata() in D:\a\1\s\src\Umbraco.Examine\LuceneIndexDiagnostics.cs:line 64
   at Umbraco.Examine.UmbracoExamineIndexDiagnostics.get_Metadata() in D:\a\1\s\src\Umbraco.Examine\UmbracoExamineIndexDiagnostics.cs:line 24
   at Umbraco.Examine.UmbracoExamineIndex.get_Metadata() in D:\a\1\s\src\Umbraco.Examine\UmbracoExamineIndex.cs:line 196
   at Umbraco.Web.Editors.ExamineManagementController.CreateModel(IIndex index) in D:\a\1\s\src\Umbraco.Web\Editors\ExamineManagementController.cs:line 193
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.OrderedEnumerable`1.<GetEnumerator>d__1.MoveNext()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs:line 677
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs:line 179
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType) in /_/Src/Newtonsoft.Json/Serialization/JsonSerializerInternalWriter.cs:line 95
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType) in /_/Src/Newtonsoft.Json/JsonSerializer.cs:line 1149
   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, Encoding effectiveEncoding)
   at System.Net.Http.Formatting.JsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, Encoding effectiveEncoding)
   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStream(Type type, Object value, Stream writeStream, HttpContent content)
   at System.Net.Http.Formatting.BaseJsonMediaTypeFormatter.WriteToStreamAsync(Type type, Object value, Stream writeStream, HttpContent content, TransportContext transportContext, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Umbraco.Web.WebApi.AngularJsonMediaTypeFormatter.<WriteToStreamAsync>d__1.MoveNext() in D:\a\1\s\src\Umbraco.Web\WebApi\AngularJsonMediaTypeFormatter.cs:line 52
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.WebHost.HttpControllerHandler.<WriteBufferedResponseContentAsync>d__22.MoveNext()

Specifics

Exceptions are shown in Umbraco backoffice Umbraco: 8.6.0 Browser: chrome, tried in other browsers Full exceptions above, screenshots: When saving: save-error When searching: search-error When accessing examine settings: examine-settings-error

Steps to reproduce

To repro exception nr 1:

Log in into Umbraco
Go to any document
Click Save and publish
the exception above appears

To repro exception nr 2:

Log in into Umbraco
Go to search (top-right search button)
type anything in the searchbox
the exception above appears

To repro exception nr 3:

Log in into Umbraco
Go to Settings -> Examine management
the exception above appears

Expected result

Exceptions are shown in Umbraco in a popup window

Actual result

There should be no exception

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 62 (30 by maintainers)

Most upvoted comments

That is interesting that you report this. We just had 2 of our sites in azure which have been fine since 8.6.1 also having this issue resurface about 45 minutes ago. Different azure subscriptions, also 8.6.1, config followed. Have been fine since my last comment here 20 days ago.

Maybe something has changed in azure environment?

fbenyah on May 26, 2020

@Shazwazza thanks for an update. So if I understand correctly there are two separate issues here:

The bug related to SqlMainDom with “load balancing”
The Lucene index becoming locked or corrupted due to Azure’s moving/changing stuff about

In our case we have two separate cases that sort of contradict themselves, both running 8.6 with recommended settings as per the (new) docs. Recently we got the case where one of the sites refused to restart (the common fix for the lucene index dying). After some debugging, it seemed the staging instance (sharing db with prod) somehow was blocking production’s restart. Stopping the slot immediately freed up the reboot to continue. This I assume would be bug 1), although I don’t quite understand why staging (receiving little to no traffic) would ever cause the db lock as it was running nicely already. As for dev/test envs in the same case, both with their own dbs, one had a working lock file, the other did not (with practically 0 interaction). I assume this to be bug 2).

The weird part, is that in the other case, we have the exact same setup, but absolutely no issues. So, I understand that its difficult since Azure is out of your control, - it is however an extremely common hosting platform. These issue affect a lot of solutions and people. We need to be able to host Umbraco on Azure, its that simple (in customers’ eyes its even simpler…). I saw your ExamineX idea and I guess that makes sense solving that part, but how much time remaining? Umbraco solutions have been unstable on Azure for years now.

JJ-DOOM on May 28, 2020

@Chris-N1 yep i’m working with @nul800sebastiaan regarding docs so we can get those changed in the interim until we can fix that. The only ‘adverse’ affects of running IgnoreLocalDb = true on your replica servers is that your site startup times can be slower because it will need to re-build all of the in-memory caches from the DB instead of from the static nucache file. However, since you are load balancing I’d assume it’s on more than one server and maybe you are just using horizontal scaling, in which case if you add a site warmup config using application initialization than azure won’t actually route anything to that node until it is ready so you won’t notice any difference. There’s lots of docs and how-tos online about this, one is here https://blog.baslijten.com/warmup-your-application-on-azure-app-service-when-scaling-up-and-swapping-slots-using-application-initialization/, you can find other Umbraco specific ones if you google something like “azure warmup application initialization umbraco”, but if you haven’t set this up before there’s a few ‘gotchas’ that you’ll discover as you read through articles, there’s no silver bullet unfortunately but would be great to have some official docs on that if anyone could help write them.

Shazwazza on Apr 29, 2020

Hi All,

Yes I think you’ve stumbled onto an issue with load balancing and this new setting. For now:

Only enable <add key="Umbraco.Core.MainDom.Lock" value="SqlMainDomLock" /> on your master server
For your front-end servers, you can use IgnoreLocalDb option listed here https://github.com/umbraco/Umbraco-CMS/issues/5035#issuecomment-566470249

MainDom controls overlapping AppDomains since .NET allows more than one app domain to run concurrently which occurs during shutdown/restart and only a single AppDomain can write to files at once. The default MainDom lock uses system-wide semaphores which don’t work on Azure so the SqlMainDomLock uses a db row as a distributed lock but the implementation currently assumes non-load balancing so that will need to be fixed.

I will update the title of this thread to reflect what the problem is.

Shazwazza on Apr 28, 2020