Umbraco-CMS: Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed

Umbraco 8.9.1

Hi!

every now and then we are seeing error logs based on requests on pages, which are using lucene indices like the member index. Its really hard to reproduce these errors, but it seems that there are still some issues with the index.

Here a sample log:

Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed
   at Lucene.Net.Index.IndexReader.EnsureOpen()
   at Lucene.Net.Index.IndexReader.Document(Int32 n)
   at Lucene.Net.Search.IndexSearcher.Doc(Int32 i)
   at Examine.LuceneEngine.LuceneSearchResults.CreateFromDocumentItem(Int32 i)
   at Examine.LuceneEngine.LuceneSearchResults.<Skip>d__22.MoveNext()
   at System.Linq.Enumerable.<TakeIterator>d__25`1.MoveNext()
   at Umbraco.Web.ExamineExtensions.ToPublishedSearchResults(IEnumerable`1 results, IPublishedSnapshot snapshot)
   at ASP._Page_Views_sitemapmember_cshtml.Execute() in C:\home\site\wwwroot\Views\sitemapmember.cshtml:line 39
   at System.Web.WebPages.WebPageBase.ExecutePageHierarchy()
   at System.Web.Mvc.WebViewPage.ExecutePageHierarchy()
   at System.Web.WebPages.WebPageBase.ExecutePageHierarchy(WebPageContext pageContext, TextWriter writer, WebPageRenderingBase startPage)
   at System.Web.Mvc.RazorView.RenderView(ViewContext viewContext, TextWriter writer, Object instance)
   at System.Web.Mvc.BuildManagerCompiledView.Render(ViewContext viewContext, TextWriter writer)
   at Umbraco.Web.Mvc.ProfilingView.Render(ViewContext viewContext, TextWriter writer)
   at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResult(ControllerContext controllerContext, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilterRecursive(IList`1 filters, Int32 filterIndex, ResultExecutingContext preContext, ControllerContext controllerContext, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_6.<BeginInvokeAction>b__4()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__1(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.<>c.<BeginExecute>b__151_2(IAsyncResult asyncResult, Controller controller)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Any ideas?

Best Dee

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 45 (21 by maintainers)

Most upvoted comments

Many thanks @timeverts

When will Examine 1.1.0 be available on the official Nuget channel?

This week

Can Examine 1.1.0 be used on the latest version of Umbraco 7?

No. A version 0.2.0 is pending. It does not include this fix but will need to be back ported, though it might not affect it in the same way since that is a different version of Lucene. I’m unsure when I’ll get that release out.

Shazwazza on Feb 16, 2021

@deekoulos I really very highly recommend that you setup load testing locally on your machine that mimics a bot crawl. That really shouldn’t be very hard, there are plenty of tools to do this. I like to use https://artillery.io/ . Replicating these issues locally is a requirement to fix things. So far I’ve been lucky to just be able to guess things for you. I would recommend checking your logs around when this error occurs for things like app restarts or for some reason if you were rebuilding indexes during this time. The TryGetIndex call checks internally in a ConcurrentDictionary. This dictionary never has items removed from it, only items added to it. The only way it cannot exist is if the index is being requested from the manager before it’s added (i.e. during startup). A full stack trace of errors is much more helpful than just the exception message.

I would also strongly recommend to not use ExamineManager.Instance and instead use Dependency Injection to inject IExamineManager.

I’m going to close this issue because the initial issue is fixed. I will release the latest Examine 1.1.0 today.

If you have further information on this latest issue/error with stack traces and steps to replicate please open a new issue with those details.

Shazwazza on Feb 12, 2021

@deekoulos

On cloud you shouldn’t worry about those settings then.

I guess the amount of requests just bring the index to the knees, when its containing large number of members.

The amount of requests to lucene should not affect bring an index down

Lucene.Net.Store.AlreadyClosedException is caused when the index reader is closed but that only happens when examine shuts down.

Only thing I can think of is that MainDom is shutting down even though your website is not actually shutting down but I have only seen that on Azure websites, not Umbraco Cloud since MainDom is controlled differently there.

You could try adding this to your serilog.config file and then monitoring when MainDom shuts down. This should always coincide with an actual app domain shutdown/restart. If you are seeing MainDom shutdown but the appdomain not actually restarting then this is the problem.

<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.MainDom" value="Debug" />
<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.MainDomSemaphoreLock" value="Debug" />
<add key="serilog:minimum-level:override:Umbraco.Core.Runtime.SqlMainDomLock" value="Debug" />

@alanwthom

Those settings look good, thank you 😃 You can also look at implementing those logging settings above to monitor MainDom shutdown vs appdomain shutdown. There’s a new 8.6.x version coming out soon with this PR fix in it: https://github.com/umbraco/Umbraco-CMS/pull/9543

This will cause MainDom to shutdown if your SQL Azure connection is severed entirely. We’ve seen this in some installs which is strange and shouldn’t happen, but alas I guess it does.

Shazwazza on Dec 15, 2020