LLamaSharp: [BUG]: KernelMemory - Simultaneous execution of AskDocument & ImportDocument

Description

System.ObjectDisposedException: Cannot access a disposed object. Object name: ‘Cannot use this SafeLLamaContextHandle - it has been disposed’. at LLama.Native.SafeLLamaContextHandle.ThrowIfDisposed() at LLama.Native.SafeLLamaContextHandle.GetLogitsIth(Int32 i) at LLama.StatelessExecutor.InferAsync(String prompt, IInferenceParams inferenceParams, CancellationToken cancellationToken)+MoveNext() at LLama.StatelessExecutor.InferAsync(String prompt, IInferenceParams inferenceParams, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at Microsoft.KernelMemory.Handlers.SummarizationHandler.SummarizeAsync(String content, IContext context) at Microsoft.KernelMemory.Handlers.SummarizationHandler.SummarizeAsync(String content, IContext context) at Microsoft.KernelMemory.Handlers.SummarizationHandler.InvokeAsync(DataPipeline pipeline, CancellationToken cancellationToken) at Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator.RunPipelineAsync(DataPipeline pipeline, CancellationToken cancellationToken) at Microsoft.KernelMemory.Pipeline.BaseOrchestrator.ImportDocumentAsync(String index, DocumentUploadRequest uploadRequest, IContext context, CancellationToken cancellationToken)

Reproduction Steps

KernelMemory - Simultaneous execution of AskDocument & ImportDocument

Environment & Configuration

  • Operating system: Windows 11
  • .NET runtime version: 8.0.8
  • LLamaSharp version: 0.16.0
  • CUDA version (if you are using cuda backend): 12
  • CPU & GPU device: CPU or GPU, Intel Core Ultra 9 & RTX 3090

Known Workarounds

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (6 by maintainers)

Most upvoted comments

Please explain why the context is created in the constructor and immediately deleted

I can’t remember why that was done, I think it was added ages ago as a hacky fix to work around an issue. I’ve been planning to overhaul the executors soon and hopefully that will get removed.

then re-created each time in InferAsync?

The StatelessExecutor creates a context for a request, uses it for that request, and then destroys it. That ensures it is completely stateless (nothing can survive between requests if the entire context is destroyed). Again, I think this could be done better now and it’ll hopefully be replaced in an overhaul soon.

As to the bug, I don’t think StatelessExecutor is safe to use concurrently (you’d probably end up with 2 requests trying to use the same context, and then when one finishes it will dispose that context, which kills the other ongoing work).