LiteDB: [BUG] Intermittent Invalid IndexPage/DataPage buffer on 0

Version Which LiteDB version/OS/.NET framework version are you using. (REQUIRED) 5.0.1. Here’s the full crash report for a load, and then a crash report for a save: Framework 4.7

LOAD (DataPage error)

Package: Acars
Version: 1.0.0.823
OS: WINDOWS
Windows: 10.0.18363
Date: 2020-02-21T02:52:10.0507405Z

Exception Stack:
LiteDB.LiteException: Invalid DataPage buffer on 0
  at LiteDB.Engine.DataPage..ctor(PageBuffer buffer)
  at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
  at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.DataService.<Read>d__5.MoveNext()
  at LiteDB.Engine.BufferReader..ctor(IEnumerable`1 source, Boolean utcDate)
  at LiteDB.Engine.DatafileLookup.Load(PageAddress rawId)
  at LiteDB.Engine.BasePipe.<LoadDocument>d__6.MoveNext()
  at LiteDB.Engine.QueryPipe.<Select>d__2.MoveNext()
  at LiteDB.Engine.QueryExecutor.<>c__DisplayClass10_0.<<ExecuteQuery>g__RunQuery|0>d.MoveNext()
  at LiteDB.BsonDataReader..ctor(IEnumerable`1 values, String collection)
  at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
  at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
  at LiteDB.LiteQueryable`1.<ToDocuments>d__26.MoveNext()
  at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
  at Acars.SimObjects.AirportDatabase.Load(String folder, SimType type)
  at Acars.SimObjects.AirportDatabase.Load()
  at Acars.MainWindow.<metroWindow_ContentRendered>d__26.MoveNext()
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.DispatcherOperation.InvokeImpl()
  at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
  at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Windows.Threading.DispatcherOperation.Invoke()
  at System.Windows.Threading.Dispatcher.ProcessQueue()
  at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
  at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
  at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
  at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
  at System.Windows.Application.RunDispatcher(Object ignore)
  at System.Windows.Application.RunInternal(Window window)
  at Acars.App.Main()

Looks like there were 9+ crashes on this one. The associated code (extra stuff removed).

lock (_lock) {
    var dbPath = GetDbName(type);

    using (var db = new LiteDatabase(dbPath)) {
        var airports = db.GetCollection<FsAirport>("airports");
        var apts = airports.FindAll();

        foreach (var apt in apts) {
            Airports.Add(apt);
        }
    }
}

Then on saving:

Package: Acars
Version: 1.0.0.821
OS: WINDOWS
Windows: 10.0.18363
Date: 2020-02-21T01:17:29.4764536Z

Exception Stack:
LiteDB.LiteException: Invalid IndexPage buffer on 0
  at LiteDB.Engine.IndexPage..ctor(PageBuffer buffer)
  at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
  at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
  at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
  at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
  at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
  at LiteDB.Engine.LiteEngine.<>c__DisplayClass30_0.<Upsert>b__0(TransactionService transaction)
  at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
  at LiteDB.LiteCollection`1.Upsert(T entity)
  at Acars.SimObjects.AirportDatabase.Save()
  at Acars.ViewModels.SettingsViewModel.<ResyncScenery>b__37_0()
  at System.Threading.Tasks.Task.Execute()
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at Acars.ViewModels.SettingsViewModel.<ResyncScenery>d__37.MoveNext()
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.DispatcherOperation.InvokeImpl()
  at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
  at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
  at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
  at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
  at System.Windows.Threading.DispatcherOperation.Invoke()
  at System.Windows.Threading.Dispatcher.ProcessQueue()
  at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
  at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
  at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
  at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
  at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
  at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
  at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
  at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
  at System.Windows.Application.RunDispatcher(Object ignore)
  at System.Windows.Application.RunInternal(Window window)
  at Acars.App.Main()

This save code is:

lock (_lock) {
    var dbFile = GetDbName(SimType);

    // Delete the old db file. Don't bother with updating rows
    if(File.Exists(dbFile)) {
        File.Delete(dbFile);
    }

    using (var db = new LiteDatabase(dbFile)) {
        var airports = db.GetCollection<FsAirport>("airports");
        var runways = db.GetCollection<FsRunway>("runways");
        var gates = db.GetCollection<FsGate>("gates");

        airports.EnsureIndex("ICAO");
        runways.EnsureIndex("ICAO");
        gates.EnsureIndex("ICAO");

        //airports.InsertBulk(Airports);
        foreach(var apt in Airports) {
            airports.Upsert(apt);

            if (apt.Runways.Count > 0) {
                runways.InsertBulk(apt.Runways);
            }

            if (apt.Gates.Count > 0) {
                gates.InsertBulk(apt.Gates);
            }
        }
    }
}

I do have this run code, but it’s hard to tell if it’s running from within this - it runs a Save() and then a Load() to switch to the newer database.


await Task.Run(() => {
    _airportDatabase.CreateDatabase(_appStateManager.Settings.SimType);

    // If they changed the SimType, reload the airport database
    if (_airportDatabase.SimType != _appStateManager.Settings.SimType) {
        _airportDatabase.Load();
    }
});

So I am deleting the old database file before saving. Maybe this isn’t the best idea. But this code hasn’t changed since using 4.x

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 2
Comments: 44 (10 by maintainers)

Most upvoted comments

@lbnascimento are you the new maintainer? Any ideas on this issue? It’s a huge problem

nabeelio on Oct 30, 2020

@botne Seeing these issues cropping up I am glad I decided to switch to Dapper+PostgreSQL eventually, LiteDB is not stable for a server workload, even if the size of data is small like my simple internal app running on t3.micro. I suppose it has it’s uses in the desktop apps though, with necessary precautions.

t00 on Oct 30, 2020

@ivanrlg, I basically put a reader lock on all read operations used, and a writer lock on all write operations used:

Example:

using (var readerLock = await _lock.ReaderLockAsync().ConfigureAwait(false))
{
	using (var stream = fileStorage.OpenRead(id))
	{
		// doing my thing with the stream
	}
}

I’ve used the AsyncReaderWriterLock from Nito

botne on Sep 20, 2020

I have a Nito.AsyncEx.AsyncReaderWriterLock in place when reading and writing files to the database. This has been stable. Tried to test yesterday (with LiteDb 5.0.8) without and got this error when I ran my unit tests. I use tasks heavily in the application.

botne on Sep 16, 2020

I managed to fix it by switching service containing LiteDatabase to a singleton so that only one instance of LiteDatabase is used for all requests. Obviously this is not perfect as a log file will be filled in with changes instead of a main db file but a delayed dispose + create with a SemaphoreSlim lock should do the trick of merging log into the main db. Assuming LiteDb log is consistent, there is no risk of data loss with this temporary “solution”.

t00 on Aug 25, 2020

From what I’ve read i think this may only be an issue when the upgrade flag is set

No. I don’t have this flag set in my application, but exception still occurs for my users.

maxkatz6 on Aug 10, 2020

Problem still exists in v5.0.9. Same error on Litedb Studio.

charly006 on Aug 6, 2020

I got a similar error message when doing multithreaded tests to see how thread safe LiteDB is

An unhandled exception of type ‘LiteDB.LiteException’ occurred in LiteDB.dll Additional information: Invalid Collection on 0. Full zero: True. Page Type: Empty. Prev/Next: 0/0. UniqueID: 5. ShareCounter: 1.

I got this error when trying to read from a collection while writing to the same collection from a different thread. Sample code that I was using is copied below.

using LiteDB;
using System;
using System.Linq;
using System.Threading;

namespace ConsoleLiteDbTest
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var database = new LiteDatabase(@"Filename=c:\test\multithreadedTests.db;"))
            {
                var readThread = new Thread(new ThreadStart(() => { readContinuously(database); }));
                var writeThread = new Thread(new ThreadStart(() => { writeContinuously(database); }));
                readThread.Start();
                writeThread.Start();
                writeThread.Join();
                readThread.Join();

                Console.WriteLine("Press any key to continue");
                Console.ReadKey();
            }
        }

        static void readContinuously(LiteDatabase database)
        {
            int recordsRead = -1;
            while (recordsRead < 100)
            {
                var collection = database.GetCollection<TestObject>("Testing");
                var records = collection.FindAll();
                var recordCount = records.Count(); //break point occurred here with the error message above.
                if (recordCount != recordsRead)
                {
                    recordsRead = recordCount;
                    Console.WriteLine($"Read in {recordsRead} records.");
                }
            }
        }

        static void writeContinuously(LiteDatabase database)
        {
            for (var i = 0; i < 100; i++)
            {
                Thread.Sleep(10);
                var collection = database.GetCollection<TestObject>("Testing");
                Console.WriteLine($"Inserting object \"{i}\"");
                collection.Insert(new TestObject { Value = $"This is object {i}" });
            }
        }

        class TestObject
        {
            public string Value { get; set; }
            public override string ToString()
            {
                return Value;
            }
        }
    }
}

trockefeller-pathway on Sep 4, 2020

Am curious if there is any update on this issue, I’ve just started to run into the same problem and may have to backout and may need to change database solutions if this is a persistent error

This was opened in February and still appears to be a problem any ideas about a timeframe for resolution

From what I’ve read i think this may only be an issue when the upgrade flag is set - I’ll do some further investigation and report if this is a problem when i switch it off

pomeara on Aug 10, 2020

I’m seeing this:

2020-04-26 21:14:28.5769|ERROR|Acars.App|UnhandledException caught : Invalid Index on 0. Full zero: True. Page Type: Empty. Prev/Next: 0/0. UniqueID: 8. ShareCounter: -1. 
2020-04-26 21:14:28.6918|ERROR|Acars.App|UnhandledException StackTrace :    at LiteDB.Engine.IndexPage..ctor(PageBuffer buffer)
   at LiteDB.Engine.BasePage.ReadPage[T](PageBuffer buffer)
   at LiteDB.Engine.Snapshot.ReadPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.Snapshot.GetPage[T](UInt32 pageID, FileOrigin& origin, Int64& position, Int32& walVersion)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass30_0.<Upsert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.LiteCollection`1.Upsert(T entity)
   at Acars.SimObjects.AirportDatabase.Save()
   at Acars.SimObjects.AirportDatabase.CreateDatabase(SimType type)
   at Acars.ViewModels.SettingsViewModel.<SyncScenery>b__37_0()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Acars.ViewModels.SettingsViewModel.<SyncScenery>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
   at MS.Internal.CulturePreservingExecutionContext.CallbackWrapper(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at MS.Internal.CulturePreservingExecutionContext.Run(CulturePreservingExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Windows.Threading.DispatcherOperation.Invoke()
   at System.Windows.Threading.Dispatcher.ProcessQueue()
   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
   at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
   at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
   at System.Windows.Application.RunDispatcher(Object ignore)
   at System.Windows.Application.RunInternal(Window window)
   at Acars.App.Main()

nabeelio on Apr 27, 2020

Oh, I just saw your PR for the missing throw, perhaps that’s why. Also getting crash reports in AppCenter

nabeelio on Apr 9, 2020