LLamaSharp: Unable to run example Project

I tried to duplicate the example project in the README, but every time i run it, i get the following error

Unhandled exception. System.TypeInitializationException: The type initializer for 'LLama.Native.NativeApi' threw an exception.
 ---> LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded. It could be one of the following reasons:
1. No LLamaSharp backend was installed. Please search LLamaSharp.Backend and install one of them.
2. You are using a device with only CPU but installed cuda backend. Please install cpu backend instead.
3. One of the dependency of the native library is missed. Please use `ldd` on linux, `dumpbin` on windows and `otool`to check if all the dependency of the native library is satisfied. Generally you could find the libraries under your output folder.
4. Try to compile llama.cpp yourself to generate a libllama library, then use `LLama.Native.NativeLibraryConfig.WithLibrary` to specify it at the very beginning of your code. For more informations about compilation, please refer to LLamaSharp repo on github.

   at LLama.Native.NativeApi..cctor()
   --- End of inner exception stack trace ---
   at LLama.Native.NativeApi.llama_max_devices()
   at LLama.Abstractions.TensorSplitsCollection..ctor()
   at LLama.Common.ModelParams..ctor(String modelPath)
   at Program.<Main>$(String[] args) in D:\projects\testing\Llama\LLamaTesting\Program.cs:line 7
   at Program.<Main>(String[] args)

D:\projects\testing\Llama\LLamaTesting\bin\x64\Debug\net8.0\LLamaTesting.exe (process 48408) exited with code -532462766.

I have LLamaSharp.Backend.Cpu installed image

Here is my code I am using

using LLama;
using LLama.Common;

string modelPath = "./models/thespis-13b-v0.5.Q5_K_S.gguf";
var prompt = "Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\r\n\r\nUser: Hello, Bob.\r\nBob: Hello. How may I help you today?\r\nUser: Please tell me the largest city in Europe.\r\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\r\nUser:"; // use the "chat-with-bob" prompt here.

var parameters = new ModelParams(modelPath)
{
    ContextSize = 1024,
    Seed = 1337,
    GpuLayerCount = 5

};
using var model = LLamaWeights.LoadFromFile(parameters);
using var context = model.CreateContext(parameters);
var executor = new InteractiveExecutor(context);

var session = new ChatSession(executor);

Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine("The chat session has started. In this example, the prompt is printed for better visual result.");
Console.ForegroundColor = ConsoleColor.White;

// show the prompt
Console.Write(prompt);
while (true)
{
    await foreach (var text in session.ChatAsync(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } }))
    {
        Console.Write(text);
    }

    Console.ForegroundColor = ConsoleColor.Green;
    prompt = Console.ReadLine();
    Console.ForegroundColor = ConsoleColor.White;
}`

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 24 (7 by maintainers)

Most upvoted comments

I have the same issue. It works with CPU, but not with GPU CUDA12. I have tested in .Net 6 and .Net 8.

LlamaSharp Version: 0.13.0

GPU: RTX 4090 CPU: i9 13900k

OS: Windows 11

"One of the dependency of the native library is missed. " What are the dependencies of the C++ DLL? Which runtime does it use?

Dumpbin gave this output:

Microsoft ® COFF/PE Dumper Version 14.39.33523.0 Copyright © Microsoft Corporation. All rights reserved.

Dump of file <omitted>\runtimes\win-x64\native\cuda12\llama.dll

File Type: DLL

Image has the following dependencies:

cudart64_12.dll
cublas64_12.dll
nvcuda.dll
KERNEL32.dll
MSVCP140.dll
VCRUNTIME140.dll
VCRUNTIME140_1.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll

Summary

   8B000 .data
    1000 .nvFatBi
 236C000 .nv_fatb
    D000 .pdata
  161000 .rdata
    1000 .reloc
    1000 .rsrc
  1C9000 .text

EDIT:

It started working after installing the NVIDIA CUDA Toolkit.

@martindevans Please reopen this issue.

Works with the dotnet-sdk-7.0 package, not with 8

If you clone this repo and run the example project does it work, or do you get the same error?

Trying now, standby