unsloth: AssertionError (assert param_data.shape == loaded_weight.shape) when serving dynamic quantized models with VLLM

I can serve unsloth dynamic quantized models just fine with VLLM when directly pulling from unsloth’s Huggingface page with following command:

vllm serve unsloth/Mistral-Small-24B-Base-2501-unsloth-bnb-4bit --dtype bfloat16 --load_format bitsandbytes --quantization bitsandbytes --max-model-len 16384

However, if I pull the dynamic quantized model from unsloth’s Huggingface, and directly push to my own Huggingface as 4bit without finetuning at all, and try to pull with vllm from my saved model, I get an error. My code to save model:

from unsloth import FastLanguageModel
import torch
max_seq_length = 2048*16 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Mistral-Small-24B-Base-2501-unsloth-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

# save 4bit version

model.push_to_hub_merged("org/my-model", tokenizer, save_method="merged_4bit_forced", private=True, token="token")

Pulling my model with vllm:

vllm serve org/my-model --dtype bfloat16 --load_format bitsandbytes --quantization bitsandbytes --max-model-len 16384

Stacktrace:

Traceback (most recent call last):
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 124, in from_engine_args
    return cls(ipc_path=ipc_path,
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 76, in __init__
    self.engine = LLMEngine(*args, **kwargs)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 273, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 52, in __init__
    self._init_executor()
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
    self.collective_rpc("load_model")
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
    return func(*args, **kwargs)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/worker/worker.py", line 183, in load_model
    self.model_runner.load_model()
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
    return loader.load_model(vllm_config=vllm_config)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1212, in load_model
    self._load_weights(model_config, model)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 1122, in _load_weights
    loaded_weights = model.load_weights(qweight_iterator)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in load_weights
    return loader.load_weights(
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
    autoloaded_weights = set(self._load_module("", self.module, weights))
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
    yield from self._load_module(prefix,
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 173, in _load_module
    loaded_params = module_load_weights(weights)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 443, in load_weights
    weight_loader(param, loaded_weight)
  File "/home/nolelin/venv310_vllm_0_7_3/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 1121, in weight_loader
    assert param_data.shape == loaded_weight.shape
AssertionError

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 12
  • Comments: 15

Most upvoted comments

Same for unsloth/gemma-3-27b-it-unsloth-bnb-4bit

This still happens. In my case when loading unsloth/gemma-3-27b-it-unsloth-bnb-4bit. I’m using vllm==0.8.2

Same here for unsloth/gemma-3-12b-it-unsloth-bnb-4bit