unsloth: AssertionError (assert param_data.shape == loaded_weight.shape) when serving dynamic quantized models with VLLM
I can serve unsloth dynamic quantized models just fine with VLLM when directly pulling from unsloth’s Huggingface page with following command:
vllm serve unsloth/Mistral-Small-24B-Base-2501-unsloth-bnb-4bit --dtype bfloat16 --load_format bitsandbytes --quantization bitsandbytes --max-model-len 16384
However, if I pull the dynamic quantized model from unsloth’s Huggingface, and directly push to my own Huggingface as 4bit without finetuning at all, and try to pull with vllm from my saved model, I get an error. My code to save model:
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048*16 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/Mistral-Small-24B-Base-2501-unsloth-bnb-4bit",
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
# save 4bit version
model.push_to_hub_merged("org/my-model", tokenizer, save_method="merged_4bit_forced", private=True, token="token")
Pulling my model with vllm:
vllm serve org/my-model --dtype bfloat16 --load_format bitsandbytes --quantization bitsandbytes --max-model-len 16384
Stacktrace:
Traceback (most recent call last):
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
return cls(ipc_path=ipc_path,
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 76, in __init__
self.engine = LLMEngine(*args, **kwargs)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 273, in __init__
self.model_executor = executor_class(vllm_config=vllm_config, )
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 52, in __init__
self._init_executor()
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
self.collective_rpc("load_model")
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
return func(*args, **kwargs)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/worker/worker.py", line 183, in load_model
self.model_runner.load_model()
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
self.model = get_model(vllm_config=self.vllm_config)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
return loader.load_model(vllm_config=vllm_config)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1212, in load_model
self._load_weights(model_config, model)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1122, in _load_weights
loaded_weights = model.load_weights(qweight_iterator)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in load_weights
return loader.load_weights(
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
autoloaded_weights = set(self._load_module("", self.module, weights))
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
yield from self._load_module(prefix,
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 173, in _load_module
loaded_params = module_load_weights(weights)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 443, in load_weights
weight_loader(param, loaded_weight)
File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 1121, in weight_loader
assert param_data.shape == loaded_weight.shape
AssertionError
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 12
- Comments: 15
Same for unsloth/gemma-3-27b-it-unsloth-bnb-4bit
This still happens. In my case when loading
unsloth/gemma-3-27b-it-unsloth-bnb-4bit. I’m using vllm==0.8.2Same here for unsloth/gemma-3-12b-it-unsloth-bnb-4bit