llm-compressor: AWQ does not actually Quant, outputs slightly bigger size file. - CoHere Model

Describe the bug Using llm_compressor, successfully calling all libraries, and quantization finishing successfully, and yet the end file/s are even bigger than the input. Either:

  1. It is failing to Quant the layers OR
  2. It’s failing to discard the original models weights when saving.

Expected behavior The output directory is a properly 4bit quanted AWQ

Environment Include all relevant environment information:

  1. OS [e.g. Ubuntu 20.04]: Ubuntu 20.04
  2. Python version [e.g. 3.7]:3.10.12
  3. LLM Compressor version or commit hash [e.g. 0.1.0, f7245c8]: 0.5.1
  4. ML framework version(s) [e.g. torch 2.3.1]: 2.5.1
  5. Other Python package versions [e.g. vLLM, compressed-tensors, numpy, ONNX]:
  6. Other relevant environment information [e.g. hardware, CUDA version]: CUDA 12.4

To Reproduce I run my script that calls llm_compressor and does the quantization

Errors No errors, the quantization finishes successfully

Additional context Script here:

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier

# Select model and load it.
MODEL_ID = "./models/CohereLabs/c4ai-command-r7b-12-2024/"

model = AutoModelForCausalLM.from_pretrained(MODEL_ID,  device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, device_map="auto", trust_remote_code=True)

# Select calibration dataset.
DATASET_ID = "mit-han-lab/pile-val-backup"
DATASET_SPLIT = "validation"

# Select number of samples. 256 samples is a good place to start.
# Increasing the number of samples can improve accuracy.
NUM_CALIBRATION_SAMPLES = 256
MAX_SEQUENCE_LENGTH = 512

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)


def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            [{"role": "user", "content": example["text"]}],
            tokenize=False,
        )
    }


ds = ds.map(preprocess)


# Tokenize inputs.
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )


# Configure the quantization algorithm to run.
recipe = [
    AWQModifier(
        model_type="custom",
        ignore=["lm_head"],
        mappings=[],
        module_mappers=[
            {
                "pattern": "re:.*.model.layers.\\d+$",
                "smooth_layer": "input_layernorm",
                "balance_layers": [
                    ("self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"),
                    ("mlp.down_proj", "mlp.gate_proj", "mlp.up_proj"),
                ],
                "to_quant_tensors": [
                    "self_attn.q_proj.weight",
                    "self_attn.k_proj.weight",
                    "self_attn.v_proj.weight",
                    "self_attn.o_proj.weight",
                    "mlp.gate_proj.weight",
                    "mlp.up_proj.weight",
                    "mlp.down_proj.weight",
                ],
            }
        ],
    )
]

# Apply algorithms.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Confirm generations of the quantized model look sane.
print("\n\n")
print("========== SAMPLE GENERATION ==============")
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to("cuda")
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0]))
print("==========================================\n\n")

# Save to disk compressed.
SAVE_DIR = "./quantized/c4ai-command-r7b-4bpw-AWQ"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 27

Commits related to this issue

Most upvoted comments

Regarding accuracy, see example in the model card here. lm_eval is a nice tool to do these evaluations, the details for running the commands are just above results table. would be interesting to see how the fp16 model compares to the AWQ model in terms of accuracy recovery

Dude, MY MAN. I knew this existed but I couldn’t find it, it’s just a simple command!! I was trying to write my own launch and test scripts. YOU ROCK. Thank you!!

Hi @phaelon74 , very cool – I did not know about local-completions to get around the multi-GPU lm_eval issue. That may be why you had to update the arc_easy task. Can you try with wikitext task? I wonder if that will work out of the box. Appreciate all the reporting!

Sooo that’s super weird. I can have a 20k context convo with this loaded LLM on my Four 3090 rig, but attempting to do the wikitxt lm-evaL TEST, it bombs me OOM. Let me see if I can spin up a bigger runpod or expand my rig to eight 3090s.

Also, I wonder, if because I was feeding it the “Use multiple GPUs” it then got real big versus running really fine tuned?

Yeah, we have updated our examples to show loading the model up on cpu and then it is sequentially onloaded to a single GPU during running of the AWQ algorithm. So it may have required that many just to load up the model, even though only one is used during AWQ. parallelizing these algorithms is an ongoing consideration

* [Is there any way quant model on multi nodes? #1382](https://github.com/vllm-project/llm-compressor/issues/1382)

This is spot on.

This was at Model Load:

Image

This was during actual quantization:

Image

This was during saving:

Image

Also, I took those pictures but never actually looked at the Wattage use during quantization. that H100 is PUTTING IN WORK!!

Regarding accuracy, see example in the model card here. lm_eval is a nice tool to do these evaluations, the details for running the commands are just above results table. would be interesting to see how the fp16 model compares to the AWQ model in terms of accuracy recovery

Great, lmk how it goes. I can look into adding the mappings specifically for the cohere models as well.

So 0.6.0 will have a lot of changes, making things much better. Does this include loading the models layers 1 at a time as opposed to all layers into VRAM?

Yes! That very recently landed on main, we wanted to get that in before the release:

* [[Performance] Sequential onloading #1263](https://github.com/vllm-project/llm-compressor/pull/1263)

Excellent. The script is running, I had to install lm_eval but now it’s cooking. Once it finishes with llama-8b-instruct, proving it works, I’ll pivot to Cohere again and try that model and see if I need to manually add mappings back in, as it relates to your script. My script most likely took parts and pieces from a lot of other scripts and I wonder if I had weird interlaced parts affecting it.

Great, lmk how it goes. I can look into adding the mappings specifically for the cohere models as well.

So 0.6.0 will have a lot of changes, making things much better. Does this include loading the models layers 1 at a time as opposed to all layers into VRAM?

Yes! That very recently landed on main, we wanted to get that in before the release: