LLamaSharp: [BUG]: KernelMemory - Simultaneous execution of AskDocument & ImportDocument
Description
System.ObjectDisposedException: Cannot access a disposed object.
Object name: ‘Cannot use this SafeLLamaContextHandle - it has been disposed’.
at LLama.Native.SafeLLamaContextHandle.ThrowIfDisposed()
at LLama.Native.SafeLLamaContextHandle.GetLogitsIth(Int32 i)
at LLama.StatelessExecutor.InferAsync(String prompt, IInferenceParams inferenceParams, CancellationToken cancellationToken)+MoveNext()
at LLama.StatelessExecutor.InferAsync(String prompt, IInferenceParams inferenceParams, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Microsoft.KernelMemory.Handlers.SummarizationHandler.SummarizeAsync(String content, IContext context)
at Microsoft.KernelMemory.Handlers.SummarizationHandler.SummarizeAsync(String content, IContext context)
at Microsoft.KernelMemory.Handlers.SummarizationHandler.InvokeAsync(DataPipeline pipeline, CancellationToken cancellationToken)
at Microsoft.KernelMemory.Pipeline.InProcessPipelineOrchestrator.RunPipelineAsync(DataPipeline pipeline, CancellationToken cancellationToken)
at Microsoft.KernelMemory.Pipeline.BaseOrchestrator.ImportDocumentAsync(String index, DocumentUploadRequest uploadRequest, IContext context, CancellationToken cancellationToken)
Reproduction Steps
KernelMemory - Simultaneous execution of AskDocument & ImportDocument
Environment & Configuration
- Operating system: Windows 11
- .NET runtime version: 8.0.8
- LLamaSharp version: 0.16.0
- CUDA version (if you are using cuda backend): 12
- CPU & GPU device: CPU or GPU, Intel Core Ultra 9 & RTX 3090
Known Workarounds
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (6 by maintainers)
I can’t remember why that was done, I think it was added ages ago as a hacky fix to work around an issue. I’ve been planning to overhaul the executors soon and hopefully that will get removed.
The
StatelessExecutorcreates a context for a request, uses it for that request, and then destroys it. That ensures it is completely stateless (nothing can survive between requests if the entire context is destroyed). Again, I think this could be done better now and it’ll hopefully be replaced in an overhaul soon.As to the bug, I don’t think
StatelessExecutoris safe to use concurrently (you’d probably end up with 2 requests trying to use the same context, and then when one finishes it will dispose that context, which kills the other ongoing work).