OrchardCore: Lucene: lock obtain timed out
Describe the bug
This happen randomly on our production server when users save a content type that are indexed by News index. Our production server is a little busy (abuot 19 of 20 GB of RAM are busy): I suppose this could be related to the issue but I hope to have some confirm about it.
This cause miss updating fo the index.
Here the stack trace:
2022-11-09 11:03:30.1740-ERROR|Vione|00-03150558224f3a55ca1ceaaf6a1d6a89-ae1974f89cb00c36-00||OrchardCore.Environment.Shell.Scope.ShellScope|Error while processing deferred task 'System.Func`2[[OrchardCore.Environment.Shell.Scope.ShellScope, OrchardCore.Abstractions, Version=1.4.0.0, Culture=neutral, PublicKeyToken=null],[System.Threading.Tasks.Task, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]' on tenant '...'. Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.................\Lucene\News\write.lock: System.IO.IOException: The process cannot access the file because another process has locked a portion of the file. : '.............\Lucene\News\write.lock'
at System.IO.Strategies.FileStreamHelpers.Lock(SafeFileHandle handle, Boolean canWrite, Int64 position, Int64 length)
at Lucene.Net.Store.NativeFSLock.Obtain()
---> System.IO.IOException: The process cannot access the file because another process has locked a portion of the file. : '.......\Lucene\News\write.lock'
at System.IO.Strategies.FileStreamHelpers.Lock(SafeFileHandle handle, Boolean canWrite, Int64 position, Int64 length)
at Lucene.Net.Store.NativeFSLock.Obtain()
--- End of inner exception stack trace ---
at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
at OrchardCore.Lucene.LuceneIndexManager.WriteAsync(String indexName, Action`1 action, Boolean close)
at OrchardCore.Lucene.LuceneIndexManager.DeleteDocumentsAsync(String indexName, IEnumerable`1 contentItemIds)
at OrchardCore.Lucene.Handlers.LuceneIndexingContentHandler.IndexingAsync(ShellScope scope, IEnumerable`1 contexts)
at OrchardCore.Environment.Shell.Scope.ShellScope.<>c__DisplayClass53_0.<<BeforeDisposeAsync>b__0>d.MoveNext() at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
at OrchardCore.Lucene.LuceneIndexManager.WriteAsync(String indexName, Action`1 action, Boolean close)
at OrchardCore.Lucene.LuceneIndexManager.DeleteDocumentsAsync(String indexName, IEnumerable`1 contentItemIds)
at OrchardCore.Lucene.Handlers.LuceneIndexingContentHandler.IndexingAsync(ShellScope scope, IEnumerable`1 contexts)
at OrchardCore.Environment.Shell.Scope.ShellScope.<>c__DisplayClass53_0.<<BeforeDisposeAsync>b__0>d.MoveNext()
To Reproduce
- have a busy server (I suppose)
- create a content type
- create a lucene index
- add the content type to the new index and to the index used in “Default search index”
Expected behavior
no lock
Thank you for every support.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (20 by maintainers)
Okay, thanks for the info.
Hmm, looking again at the stack trace, I see twice
Lucene.Net.Index.IndexWriter..ctor, so first it would mean that you have 2 search indexes related to the content type of the item that is published, right?Then, because for a given index we build a writer once, it would mean at least that the tenant was rebuild (for example after updating some admin settings), or that the application was re-started, not necessarily just before the failing publish but since the previous one.
So still possible that a writer was not well disposed, maybe we would need to check the underlying lucene lock more in depth, for example by using
IndexWriter.IsLocked(directory). Hmm as I remember I suggested such things in an old PR where I wanted to make it working even if multiple instances was sharing the same file system.In the meantime for info I let you read the writer dispose summary comments where we see
that this may be a costly operation, an use case wherethe write lock will still be held, howto force the write lock to be released (dangerous)and so on. Also see the last note in case of aOutOfMemoryException.