vllm: AWQ + Marlin Error
I convert model follow AutoAWQ library as follow script.
- Quantize with Marlin
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = 'mistralai/Mistral-7B-Instruct-v0.2'
quant_path = 'mistral-instruct-v0.2-awq-marlin'
quant_config = { "zero_point": False, "q_group_size": 128, "w_bit": 4, "version": "Marlin" }
# Load model
model = AutoAWQForCausalLM.from_pretrained(
model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Quantize
model.quantize(tokenizer, quant_config=quant_config)
# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
print(f'Model is quantized and saved at "{quant_path}"')
- Generate
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer
quant_path = "./mistral-instruct-v0.2-awq-marlin"
# Load model
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=False)
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
# Convert prompt to tokens
prompt_template = "[INST] {prompt} [/INST]"
prompt = "You're standing on the surface of the Earth. "\
"You walk one mile south, one mile west and one mile north. "\
"You end up exactly where you started. Where are you?"
tokens = tokenizer(
prompt_template.format(prompt=prompt),
return_tensors='pt'
).input_ids.cuda()
# Generate output
generation_output = model.generate(
tokens,
streamer=streamer,
max_new_tokens=512
)
2 steps above work perfectly.
But when I using that quantization folder to start serving in vllm with this script.
python services/vllm/api_server.py \
--host 0.0.0.0 \
--port 8000 \
--disable-log-requests \
--model /root/AutoAWQ/mistral-instruct-v0.2-awq-marlin \
--quantization awq \
--trust-remote-code \
--dtype float16 \
The error show that:
File "/root/vllm/vllm/engine/llm_engine.py", line 101, in __init__
self.model_executor = executor_class(model_config, cache_config,
File "/root/vllm/vllm/executor/gpu_executor.py", line 42, in __init__
self._init_worker()
File "/root/vllm/vllm/executor/gpu_executor.py", line 77, in _init_worker
self.driver_worker.load_model()
File "/root/vllm/vllm/worker/worker.py", line 99, in load_model
self.model_runner.load_model()
File "/root/vllm/vllm/worker/model_runner.py", line 89, in load_model
self.model = get_model(self.model_config,
File "/root/vllm/vllm/model_executor/utils.py", line 52, in get_model
return get_model_fn(model_config, device_config, **kwargs)
File "/root/vllm/vllm/model_executor/model_loader.py", line 86, in get_model
model.load_weights(model_config.model, model_config.download_dir,
File "/root/vllm/vllm/model_executor/models/llama.py", line 391, in load_weights
weight_loader(param, loaded_weight)
File "/root/vllm/vllm/model_executor/layers/linear.py", line 556, in weight_loader
loaded_weight = loaded_weight.narrow(input_dim, start_idx,
RuntimeError: start (0) + length (14336) exceeds dimension size (896).
It look like the weight can not load and vllm did not support AWQ with Marlin kernel properly. How can we fix this error. Thank you.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 16 (4 by maintainers)
@hllj
Loading AWQ models saved in Marlin is not currently supported. I was not aware that AWQ models could be converted to Marlin format. I will look into adding support for this
Can you post that model to Hugging Face hub? This would make it easier for me
cc @alexm-nm @simon-mo @mgoin
AWQ has many versions at this point. I would recommend using gemv_fast from AutoAWQ as it’s faster than Marlin and exllamav2 in decoding. However, it’s not implemented yet, although I laid out some ground work in a PR
Got it - so you do the repacking on the fly (during
from_linearwhich is called when the model is being created) as opposed to serializing the model in theGEMVFastformat?