vllm: AWQ + Marlin Error

I convert model follow AutoAWQ library as follow script.

  1. Quantize with Marlin
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
quant_path = 'mistral-instruct-v0.2-awq-marlin'
quant_config = { "zero_point": False, "q_group_size": 128, "w_bit": 4, "version": "Marlin" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(
    model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize
model.quantize(tokenizer, quant_config=quant_config)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')
  1. Generate
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer


quant_path = "./mistral-instruct-v0.2-awq-marlin"

# Load model
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=False)
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = "[INST] {prompt} [/INST]"

prompt = "You're standing on the surface of the Earth. "\
        "You walk one mile south, one mile west and one mile north. "\
        "You end up exactly where you started. Where are you?"

tokens = tokenizer(
    prompt_template.format(prompt=prompt), 
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=512
)

2 steps above work perfectly.

But when I using that quantization folder to start serving in vllm with this script.

python services/vllm/api_server.py \
    --host 0.0.0.0 \
    --port 8000 \
    --disable-log-requests \
    --model /root/AutoAWQ/mistral-instruct-v0.2-awq-marlin \
    --quantization awq \
    --trust-remote-code \
    --dtype float16 \

The error show that:

  File "/root/vllm/vllm/engine/llm_engine.py", line 101, in __init__
    self.model_executor = executor_class(model_config, cache_config,
  File "/root/vllm/vllm/executor/gpu_executor.py", line 42, in __init__
    self._init_worker()
  File "/root/vllm/vllm/executor/gpu_executor.py", line 77, in _init_worker
    self.driver_worker.load_model()
  File "/root/vllm/vllm/worker/worker.py", line 99, in load_model
    self.model_runner.load_model()
  File "/root/vllm/vllm/worker/model_runner.py", line 89, in load_model
    self.model = get_model(self.model_config,
  File "/root/vllm/vllm/model_executor/utils.py", line 52, in get_model
    return get_model_fn(model_config, device_config, **kwargs)
  File "/root/vllm/vllm/model_executor/model_loader.py", line 86, in get_model
    model.load_weights(model_config.model, model_config.download_dir,
  File "/root/vllm/vllm/model_executor/models/llama.py", line 391, in load_weights
    weight_loader(param, loaded_weight)
  File "/root/vllm/vllm/model_executor/layers/linear.py", line 556, in weight_loader
    loaded_weight = loaded_weight.narrow(input_dim, start_idx,
RuntimeError: start (0) + length (14336) exceeds dimension size (896).

It look like the weight can not load and vllm did not support AWQ with Marlin kernel properly. How can we fix this error. Thank you.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 16 (4 by maintainers)

Most upvoted comments

@hllj

Loading AWQ models saved in Marlin is not currently supported. I was not aware that AWQ models could be converted to Marlin format. I will look into adding support for this

Can you post that model to Hugging Face hub? This would make it easier for me

cc @alexm-nm @simon-mo @mgoin

AWQ has many versions at this point. I would recommend using gemv_fast from AutoAWQ as it’s faster than Marlin and exllamav2 in decoding. However, it’s not implemented yet, although I laid out some ground work in a PR

Got it - so you do the repacking on the fly (during from_linear which is called when the model is being created) as opposed to serializing the model in the GEMVFast format?