vllm: KeyError: 'base_model.model.model.layers.0.self_attn.qkv_proj.lora_A.weight'
When I try to inference my finetuned code llama model using vllm, getting this error
File "/usr/local/lib/python3.9/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method return executor(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker.py", line 70, in init_model self.model = get_model(self.model_config) File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/model_loader.py", line 103, in get_model model.load_weights(model_config.model, model_config.download_dir, File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/models/llama.py", line 367, in load_weights param = state_dict[name.replace(weight_name, "qkv_proj")] KeyError: 'base_model.model.model.layers.0.self_attn.qkv_proj.lora_A.weight'
Some Shape Of my models
`<html>
| model.layers.0.input_layernorm.weight | [4096] | BF16 |
|---|---|---|
| model.layers.0.mlp.down_proj.weight | [4096,11008] | BF16 |
| model.layers.0.mlp.gate_proj.weight | [11008,4096] | BF16 |
| model.layers.0.mlp.up_proj.weight | [11008,4096] | BF16 |
| model.layers.0.post_attention_layernorm.weight | [4096] | BF16 |
| model.layers.0.self_attn.k_proj.weight | [4096,4096] | BF16 |
| model.layers.0.self_attn.o_proj.weight | [4096,4096] | BF16 |
| model.layers.0.self_attn.q_proj.weight | [4096,4096] | BF16 |
| model.layers.0.self_attn.v_proj.weight | [4096,4096] | BF16 |
Any Suggestion or help is highly appreciated.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22
still does not work.
I have tried
mistralai/Mistral-7B-Instruct-v0.3andmistralai/Ministral-8B-Instruct-2410here is the code:
here is the error:
The supported targets are only limited to
Currently vLLM does not support merging LoRA weights. Hence the model loader is erroring. Contribution strongly welcomed here! Ideally you can apply the LoRA weights automatically on the model loading process. I believe this PR does what you want: https://github.com/vllm-project/vllm/pull/289
I’m closing this PR in favor of #182