llm-compressor: AWQ does not actually Quant, outputs slightly bigger size file. - CoHere Model
Describe the bug Using llm_compressor, successfully calling all libraries, and quantization finishing successfully, and yet the end file/s are even bigger than the input. Either:
- It is failing to Quant the layers OR
- It’s failing to discard the original models weights when saving.
Expected behavior The output directory is a properly 4bit quanted AWQ
Environment Include all relevant environment information:
- OS [e.g. Ubuntu 20.04]: Ubuntu 20.04
- Python version [e.g. 3.7]:3.10.12
- LLM Compressor version or commit hash [e.g. 0.1.0,
f7245c8]: 0.5.1 - ML framework version(s) [e.g. torch 2.3.1]: 2.5.1
- Other Python package versions [e.g. vLLM, compressed-tensors, numpy, ONNX]:
- Other relevant environment information [e.g. hardware, CUDA version]: CUDA 12.4
To Reproduce I run my script that calls llm_compressor and does the quantization
Errors No errors, the quantization finishes successfully
Additional context Script here:
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier
# Select model and load it.
MODEL_ID = "./models/CohereLabs/c4ai-command-r7b-12-2024/"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, device_map="auto", trust_remote_code=True)
# Select calibration dataset.
DATASET_ID = "mit-han-lab/pile-val-backup"
DATASET_SPLIT = "validation"
# Select number of samples. 256 samples is a good place to start.
# Increasing the number of samples can improve accuracy.
NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 512
# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)
def preprocess(example):
return {
"text": tokenizer.apply_chat_template(
[{"role": "user", "content": example["text"]}],
tokenize=False,
)
}
ds = ds.map(preprocess)
# Tokenize inputs.
def tokenize(sample):
return tokenizer(
sample["text"],
padding=False,
max_length=MAX_SEQUENCE_LENGTH,
truncation=True,
add_special_tokens=False,
)
# Configure the quantization algorithm to run.
recipe = [
AWQModifier(
model_type="custom",
ignore=["lm_head"],
mappings=[],
module_mappers=[
{
"pattern": "re:.*.model.layers.\\d+$",
"smooth_layer": "input_layernorm",
"balance_layers": [
("self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"),
("mlp.down_proj", "mlp.gate_proj", "mlp.up_proj"),
],
"to_quant_tensors": [
"self_attn.q_proj.weight",
"self_attn.k_proj.weight",
"self_attn.v_proj.weight",
"self_attn.o_proj.weight",
"mlp.gate_proj.weight",
"mlp.up_proj.weight",
"mlp.down_proj.weight",
],
}
],
)
]
# Apply algorithms.
oneshot(
model=model,
dataset=ds,
recipe=recipe,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)
# Confirm generations of the quantized model look sane.
print("\n\n")
print("========== SAMPLE GENERATION ==============")
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0]))
print("==========================================\n\n")
# Save to disk compressed.
SAVE_DIR = "./quantized/c4ai-command-r7b-4bpw-AWQ"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 27
Commits related to this issue
- AWQ Cohere, Mistral & Gemma mappings (#1570) SUMMARY: Add AWQ mappings for `CohereForCausalLM` models, which don't have a post_attention_layernorm and instead run MLP and self_attn computations in pa... — committed to vllm-project/llm-compressor by brian-dellabetta a year ago
- AWQ Cohere, Mistral & Gemma mappings (#1570) SUMMARY: Add AWQ mappings for `CohereForCausalLM` models, which don't have a post_attention_layernorm and instead run MLP and self_attn computations in pa... — committed to vllm-project/llm-compressor by brian-dellabetta a year ago
Dude, MY MAN. I knew this existed but I couldn’t find it, it’s just a simple command!! I was trying to write my own launch and test scripts. YOU ROCK. Thank you!!
Sooo that’s super weird. I can have a 20k context convo with this loaded LLM on my Four 3090 rig, but attempting to do the wikitxt lm-evaL TEST, it bombs me OOM. Let me see if I can spin up a bigger runpod or expand my rig to eight 3090s.
This is spot on.
This was at Model Load:
This was during actual quantization:
This was during saving:
Also, I took those pictures but never actually looked at the Wattage use during quantization. that H100 is PUTTING IN WORK!!
Regarding accuracy, see example in the model card here.
lm_evalis a nice tool to do these evaluations, the details for running the commands are just above results table. would be interesting to see how the fp16 model compares to the AWQ model in terms of accuracy recoveryExcellent. The script is running, I had to install lm_eval but now it’s cooking. Once it finishes with llama-8b-instruct, proving it works, I’ll pivot to Cohere again and try that model and see if I need to manually add mappings back in, as it relates to your script. My script most likely took parts and pieces from a lot of other scripts and I wonder if I had weird interlaced parts affecting it.
Great, lmk how it goes. I can look into adding the mappings specifically for the cohere models as well.
Yes! That very recently landed on main, we wanted to get that in before the release: