OrchardCore: Lucene: lock obtain timed out

Describe the bug

This happen randomly on our production server when users save a content type that are indexed by News index. Our production server is a little busy (abuot 19 of 20 GB of RAM are busy): I suppose this could be related to the issue but I hope to have some confirm about it.

This cause miss updating fo the index.

Here the stack trace:

2022-11-09 11:03:30.1740-ERROR|Vione|00-03150558224f3a55ca1ceaaf6a1d6a89-ae1974f89cb00c36-00||OrchardCore.Environment.Shell.Scope.ShellScope|Error while processing deferred task 'System.Func`2[[OrchardCore.Environment.Shell.Scope.ShellScope, OrchardCore.Abstractions, Version=1.4.0.0, Culture=neutral, PublicKeyToken=null],[System.Threading.Tasks.Task, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]' on tenant '...'. Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@.................\Lucene\News\write.lock: System.IO.IOException: The process cannot access the file because another process has locked a portion of the file. : '.............\Lucene\News\write.lock'
   at System.IO.Strategies.FileStreamHelpers.Lock(SafeFileHandle handle, Boolean canWrite, Int64 position, Int64 length)
   at Lucene.Net.Store.NativeFSLock.Obtain()
 ---> System.IO.IOException: The process cannot access the file because another process has locked a portion of the file. : '.......\Lucene\News\write.lock'
   at System.IO.Strategies.FileStreamHelpers.Lock(SafeFileHandle handle, Boolean canWrite, Int64 position, Int64 length)
   at Lucene.Net.Store.NativeFSLock.Obtain()
   --- End of inner exception stack trace ---
   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
   at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
   at OrchardCore.Lucene.LuceneIndexManager.WriteAsync(String indexName, Action`1 action, Boolean close)
   at OrchardCore.Lucene.LuceneIndexManager.DeleteDocumentsAsync(String indexName, IEnumerable`1 contentItemIds)
   at OrchardCore.Lucene.Handlers.LuceneIndexingContentHandler.IndexingAsync(ShellScope scope, IEnumerable`1 contexts)
   at OrchardCore.Environment.Shell.Scope.ShellScope.<>c__DisplayClass53_0.<<BeforeDisposeAsync>b__0>d.MoveNext()    at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
   at Lucene.Net.Index.IndexWriter..ctor(Directory d, IndexWriterConfig conf)
   at OrchardCore.Lucene.LuceneIndexManager.WriteAsync(String indexName, Action`1 action, Boolean close)
   at OrchardCore.Lucene.LuceneIndexManager.DeleteDocumentsAsync(String indexName, IEnumerable`1 contentItemIds)
   at OrchardCore.Lucene.Handlers.LuceneIndexingContentHandler.IndexingAsync(ShellScope scope, IEnumerable`1 contexts)
   at OrchardCore.Environment.Shell.Scope.ShellScope.<>c__DisplayClass53_0.<<BeforeDisposeAsync>b__0>d.MoveNext()

To Reproduce

  1. have a busy server (I suppose)
  2. create a content type
  3. create a lucene index
  4. add the content type to the new index and to the index used in “Default search index”

Expected behavior

no lock

Thank you for every support.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 20 (20 by maintainers)

Most upvoted comments

Okay, thanks for the info.

Hmm, looking again at the stack trace, I see twice Lucene.Net.Index.IndexWriter..ctor, so first it would mean that you have 2 search indexes related to the content type of the item that is published, right?

Then, because for a given index we build a writer once, it would mean at least that the tenant was rebuild (for example after updating some admin settings), or that the application was re-started, not necessarily just before the failing publish but since the previous one.

So still possible that a writer was not well disposed, maybe we would need to check the underlying lucene lock more in depth, for example by using IndexWriter.IsLocked(directory). Hmm as I remember I suggested such things in an old PR where I wanted to make it working even if multiple instances was sharing the same file system.

In the meantime for info I let you read the writer dispose summary comments where we see that this may be a costly operation, an use case where the write lock will still be held, how to force the write lock to be released (dangerous) and so on. Also see the last note in case of a OutOfMemoryException.

    /// <summary>
    /// Commits all changes to an index, waits for pending merges
    /// to complete, and closes all associated files.
    /// <para/>
    /// This is a "slow graceful shutdown" which may take a long time
    /// especially if a big merge is pending: If you only want to close
    /// resources use <see cref="Rollback()"/>. If you only want to commit
    /// pending changes and close resources see <see cref="Dispose(bool)"/>.
    /// <para/>
    /// Note that this may be a costly
    /// operation, so, try to re-use a single writer instead of
    /// closing and opening a new one.  See <see cref="Commit()"/> for
    /// caveats about write caching done by some IO devices.
    ///
    /// <para> If an <see cref="Exception"/> is hit during close, eg due to disk
    /// full or some other reason, then both the on-disk index
    /// and the internal state of the <see cref="IndexWriter"/> instance will
    /// be consistent.  However, the close will not be complete
    /// even though part of it (flushing buffered documents)
    /// may have succeeded, so the write lock will still be
    /// held.</para>
    ///
    /// <para> If you can correct the underlying cause (eg free up
    /// some disk space) then you can call <see cref="Dispose()"/> again.
    /// Failing that, if you want to force the write lock to be
    /// released (dangerous, because you may then lose buffered
    /// docs in the <see cref="IndexWriter"/> instance) then you can do
    /// something like this:</para>
    ///
    /// <code>
    /// try 
    /// {
    ///     writer.Dispose();
    /// } 
    /// finally 
    /// {
    ///     if (IndexWriter.IsLocked(directory)) 
    ///     {
    ///         IndexWriter.Unlock(directory);
    ///     }
    /// }
    /// </code>
    /// 
    /// after which, you must be certain not to use the writer
    /// instance anymore.
    ///
    /// <para><b>NOTE</b>: if this method hits an <see cref="OutOfMemoryException"/>
    /// you should immediately dispose the writer, again.  See 
    /// <see cref="IndexWriter"/> for details.</para>
    /// </summary>
    /// <exception cref="IOException"> if there is a low-level IO error </exception>
    [MethodImpl(MethodImplOptions.NoInlining)]
    public void Dispose()