vllm: KeyError: 'base_model.model.model.layers.0.self_attn.qkv_proj.lora_A.weight'

When I try to inference my finetuned code llama model using vllm, getting this error File "/usr/local/lib/python3.9/dist-packages/vllm/engine/ray_utils.py", line 32, in execute_method return executor(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/vllm/worker/worker.py", line 70, in init_model self.model = get_model(self.model_config) File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/model_loader.py", line 103, in get_model model.load_weights(model_config.model, model_config.download_dir, File "/usr/local/lib/python3.9/dist-packages/vllm/model_executor/models/llama.py", line 367, in load_weights param = state_dict[name.replace(weight_name, "qkv_proj")] KeyError: 'base_model.model.model.layers.0.self_attn.qkv_proj.lora_A.weight' vllm Some Shape Of my models `<html>

<body>
model.layers.0.input_layernorm.weight [4096] BF16
model.layers.0.mlp.down_proj.weight [4096,11008] BF16
model.layers.0.mlp.gate_proj.weight [11008,4096] BF16
model.layers.0.mlp.up_proj.weight [11008,4096] BF16
model.layers.0.post_attention_layernorm.weight [4096] BF16
model.layers.0.self_attn.k_proj.weight [4096,4096] BF16
model.layers.0.self_attn.o_proj.weight [4096,4096] BF16
model.layers.0.self_attn.q_proj.weight [4096,4096] BF16
model.layers.0.self_attn.v_proj.weight [4096,4096] BF16
</body> </html>`

Any Suggestion or help is highly appreciated.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22

Most upvoted comments

still does not work.

I have tried mistralai/Mistral-7B-Instruct-v0.3 and mistralai/Ministral-8B-Instruct-2410

here is the code:

llm = LLM(model= args.llm_name, dtype='bfloat16', max_model_len= max_len, 
            tensor_parallel_size=torch.cuda.device_count(), gpu_memory_utilization= args.gpu_memory_utilization, #seed=None,
            trust_remote_code=True,
            quantization="bitsandbytes", load_format="bitsandbytes", enforce_eager=True, 
            enable_lora=enable_lora,
            tokenizer_mode= "mistral" if args.llm_name.startswith('mistralai') else 'auto'
        )   

here is the error:

Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:00<00:02, 1.34it/s] Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:01<00:01, 1.27it/s] [rank0]: Traceback (most recent call last): [rank0]: File “/home/ubuntu/moa/eval.py”, line 169, in <module> [rank0]: llm = LLM(model= args.llm_name, dtype=‘bfloat16’, max_model_len= max_len, [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/entrypoints/llm.py”, line 177, in init [rank0]: self.llm_engine = LLMEngine.from_engine_args( [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py”, line 573, in from_engine_args [rank0]: engine = cls( [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py”, line 334, in init [rank0]: self.model_executor = executor_class( [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/executor/executor_base.py”, line 47, in init [rank0]: self._init_executor() [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/executor/gpu_executor.py”, line 40, in _init_executor [rank0]: self.driver_worker.load_model() [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py”, line 183, in load_model [rank0]: self.model_runner.load_model() [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/worker/model_runner.py”, line 1058, in load_model [rank0]: self.model = get_model(model_config=self.model_config, [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader/init.py”, line 19, in get_model [rank0]: return loader.load_model(model_config=model_config, [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py”, line 1148, in load_model [rank0]: self._load_weights(model_config, model) [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py”, line 1082, in _load_weights [rank0]: model.load_weights(qweight_iterator) [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/llama.py”, line 582, in load_weights [rank0]: loader.load_weights( [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/utils.py”, line 203, in load_weights [rank0]: autoloaded_weights = list(self._load_module(“”, self.module, weights)) [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/utils.py”, line 182, in _load_module [rank0]: yield from self._load_module(prefix, [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/utils.py”, line 169, in _load_module [rank0]: module_load_weights(weights) [rank0]: File “/opt/conda/lib/python3.10/site-packages/vllm/model_executor/models/llama.py”, line 397, in load_weights [rank0]: param = params_dict[name] [rank0]: KeyError: ‘layers.0.self_attn.qkv_proj.weight’

Hello everyone. Can I get help? I’m using inference on phi2 customed model and for inference I’m using MODAL and VLLM and I’m getting : base_model.model.model.layers.0.mlp.fc2.lora_A.weight @Soumendraprasad @SuperBruceJia

The supported targets are only limited to

target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
        ],

Currently vLLM does not support merging LoRA weights. Hence the model loader is erroring. Contribution strongly welcomed here! Ideally you can apply the LoRA weights automatically on the model loading process. I believe this PR does what you want: https://github.com/vllm-project/vllm/pull/289

I’m closing this PR in favor of #182