WasmEdge: bug: Can not run llama-api-server with kuasar

Summary

I’m trying to run llama-api-server with Llama-3-8B in wasm-sandboxer of Kuasar! But got some errors.

Current State

No response

Expected State

No response

Reproduction steps

  1. Load wasmedge plugins, which are already downloaded and installed in my computer
        PluginManager::load(None).unwrap();
        let mut host_options = HostRegistrationConfigOptions::default();
        host_options = host_options.wasi(true);
        let config = ConfigBuilder::new(CommonConfigOptions::default())
            .with_host_registration_config(host_options)
            .build()
            .unwrap();
        let vm = VmBuilder::new()
            .with_config(config)
            .build()
            .map_err(anyhow::Error::msg)
            .unwrap();
  1. Initialize a wasi instance:
    let wasi_instance: &mut WasiInstance = vm.wasi_module_mut().ok_or(RunError::NoModule)?;
    wasi_instance.initialize(
        Some(args.iter().map(|s| s as &str).collect()),
        Some(envs.iter().map(|s| s as &str).collect()),
        Some(preopens.iter().map(|s| s as &str).collect()),
    );

args: ["llama-api-server.wasm", "--prompt-template", "llama-3-chat", "--ctx-size 4096", "--model-name", "Llama-3-8B", "--log-all"] preopens: ["/:/run/containerd/io.containerd.runtime.v2.task/k8s.io/93f3bfa0c945317105ea245062e59a6d26890b96ce9165e4dfc4aa40a6a86ccd/rootfs"] and /run/containerd/io.containerd.runtime.v2.task/k8s.io/xxxxx/rootfs is the rootfs path of a container:

/run/containerd/io.containerd.runtime.v2.task/k8s.io/af59f9e042cc72bcedd76360cde7b97666c60e2de6ad1d70ae494232d8896ab1/rootfs:
total 5608312
drwxr-xr-x 5 root root       4096 Mar 25 15:17 chatbot-ui
-rw-r--r-- 1 root root    6895613 Jul 27 16:28 llama-api-server.wasm
-rw-r--r-- 1 root root    3010900 Jul 27 16:37 llama-chat.wasm
-rw-r--r-- 1 root root 5732987104 Jul 27 16:34 Meta-Llama-3-8B-Instruct-Q5_K_M.gguf
  1. Initialize the wasi_nn plug-in with the preloads
        let nn_preload = "default:GGML:AUTO:/run/containerd/io.containerd.runtime.v2.task/k8s.io/af59f9e042cc72bcedd76360cde7b97666c60e2de6ad1d70ae494232d8896ab1/rootfs/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf";
        PluginManager::nn_preload(vec![
            wasmedge_sdk::plugin::NNPreload::from_str(&nn_preload).unwrap()
        ]);
  1. Registry module and run the function
    let vm = vm
        .register_module_from_file("main", "/run/containerd/io.containerd.runtime.v2.task/k8s.io/af59f9e042cc72bcedd76360cde7b97666c60e2de6ad1d70ae494232d8896ab1/rootfs/llama-api-server.wasm")
        .map_err(RunError::WasmEdge)?;
    vm.run_func(Some("main"), "_start", params!())
        .map_err(RunError::WasmEdge)?;

Screenshots

Any logs you want to share for showing the specific issue

Finally, I got this error

[2024-07-27 17:49:11.684] [error] instantiation failed: unknown import, Code: 0x62
[2024-07-27 17:49:11.684] [error]     When linking module: "wasi_ephemeral_nn" , function name: "set_input"
[2024-07-27 17:49:11.684] [error]     At AST node: import description
[2024-07-27 17:49:11.684] [error]     At AST node: import section
[2024-07-27 17:49:11.684] [error]     At AST node: module

Components

Rust SDK

WasmEdge Version or Commit you used

wasmEdge lib 0.13.5, wasmedge-sdk 0.13.2

Operating system information

CentOS-like OS

Hardware Architecture

x86_64

Compiler flags and options

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 19 (6 by maintainers)

Most upvoted comments

I got it. And I also add “wasi_logging” plugin:

    let mut vm = VmBuilder::new()
        .with_config(config)
        .with_plugin_wasi_nn()
        .with_plugin("wasi_logging", None)
        .build()
        .unwrap();

But the result is also failed:

[2024-07-29 11:27:19.633] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:131: server version: 0.12.5                                                      
[2024-07-29 11:27:19.633] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:139: model_name: Llama-3-8B                                                      
[2024-07-29 11:27:19.633] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:148: model_alias: default                                                        
[2024-07-29 11:27:19.633] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:167: ctx_size: 4096                                                              
[2024-07-29 11:27:19.633] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:186: batch_size: 512                                                             
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:200: prompt_template: llama-3-chat                                               
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:213: n_predict: 1024                                                             
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:216: n_gpu_layers: 100                                                           
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:227: temp: 1                                                                     
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:230: top_p: 1                                                                    
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:233: repeat_penalty: 1.1                                                         
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:236: presence_penalty: 0                                                         
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] server_config: llama_api_server in llama-api-server/src/main.rs:239: frequency_penalty: 0                                                        
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [info] llama-core: llama_core in llama-core/src/lib.rs:417: Initializing the core context                                                               
[2024-07-29 11:27:19.634] [wasi_logging_stdout] [error] llama-core: llama_core in llama-core/src/lib.rs:285: Backend Error: WASI-NN Backend Error: Not Found                                            
Error: Operation("Backend Error: WASI-NN Backend Error: Not Found")