unsloth: Applying LoRA Doesn't Change Model Output

Greetings,

I am extremely confused why my model generates consistent result w/o my rank=64 LoRA. What’s even more confusing is, the LoRA works in my notebook after training. But whenn I start fresh, loading the LoRA again, it gives the output of the original model.

model_name=="Qwen/Qwen2.5-1.5B-Instruct",
vllm==0.7.3
trl==0.15.2
unsloth==2025.2.14
unsloth_zoo==2025.2.7

So I followed the tutorial notebook, trained the model using GRPO trainer. Then within the notebook, I model.save_lora("grpo_saved_lora"), then I do this using vllm with greedy decoding:

sampling_params = SamplingParams(
    max_tokens=32,  # Maximum number of tokens to generate
    temperature=0,  # Temperature of 0 means no randomness (deterministic)
    top_p=1.0,      # Consider all tokens in the distribution
    top_k=1         # Only select the most likely token at each step
)

outputs = model.fast_generate(
    test_split,
    sampling_params = sampling_params,
    lora_request = model.load_lora("grpo_saved_lora"),
)

These are all fine, I have different set of output compare with before training. But then, when I try to load the model lora with a separate script. The output is suddenly consistent with the original model, even I loaded the lora in this way.

cfg = load_config(args.config)
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name=cfg.model_name,
        max_seq_length=cfg.max_seq_length,
        load_in_4bit=cfg.load_in_4bit,
        fast_inference=cfg.fast_inference,
        max_lora_rank=cfg.lora_rank,
        gpu_memory_utilization=cfg.gpu_memory_utilization,
    )
    lora = None
    
    if args.lora_path is not None:
        model = FastLanguageModel.get_peft_model(
            model,
            r=cfg.lora_rank,
            target_modules=cfg.target_modules,
            lora_alpha=cfg.lora_rank,
            use_gradient_checkpointing="unsloth",
            random_state=3407,
        )
        lora = model.load_lora(args.lora_path)
        print('using lora ' + args.lora_path)
            
    _, _, test_dataset = get_dataset("ecqa")
    
    sampling_params = SamplingParams( # Greedy decoding
        max_tokens=32,  # Maximum number of tokens to generate
        temperature=0,  # Temperature of 0 means no randomness (deterministic)
        top_p=1.0,      # Consider all tokens in the distribution
        top_k=1         # Only select the most likely token at each step
    )
    
    vllm_prompts = [x['prompt'] for x in test_dataset]
    
    outputs = model.fast_generate(
        vllm_prompts,
        sampling_params = sampling_params,
        lora_request = lora
    )

Could you please help me? I am really confused.

Thank you very much.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 51 (4 by maintainers)

Most upvoted comments

Hi apologies everyone - let me investigate and I’ll see what I can do to fix the issue!

You are not alone, it is related to #1521

So merging LoRA to the base model is an issue, and even for those who successfully merge and try to inference the merged model, they still face an issue where performance deteriorates ???

Yes. I have spent hundreds of dollars on A100 in order to finetune with GRPO Phi-4 and now that model is stuck on the virtual runtime because I cannot merge the model and push it to HF, and neither pushing only the lora works, because applying that lora gives only vanilla outputs.

I also tried retraining with latest versions of unsloth using smaller models (phi-4-mini and gemma-3), also using their own colabs without touching anything. Nothing. For now, it is a waste of money, I’m sorry to say that.

I think the problem is not fixed yet. I reloaded an already done training as suggested but the saved lora and the hf pushed ones have the same behaviour of the original model. I’m on “unsloth==2025.3.18” “unsloth-zoo==2025.3.16”

The wird thing is that with the previous version of unlosth (2025.3.13) the saved lora worked, the merged and pushed model not.

The training code:

[...]

model_id = "unsloth/phi-4"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = model_id,
    max_seq_length = 8000,
    load_in_4bit = True, 
    fast_inference = True, 
    max_lora_rank = lora_rank,
    gpu_memory_utilization = gpu_memory_utilization, 
    cache_dir = cache_dir
)

model = FastLanguageModel.get_peft_model(
    model,
    r = lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ], # Remove QKVO if out of memory
    lora_alpha = lora_rank,
    use_gradient_checkpointing = "unsloth", 
    random_state = 42
)

[...]

trainer.train()

# tried all of these saving methods
##model.save_pretrained(f"{checkpoint_dir}/grpo_saved_lora_fix")  
##tokenizer.save_pretrained(f"{checkpoint_dir}/grpo_saved_lora_fix")
model.save_lora(f"{checkpoint_dir}/grpo_saved_lora_fix") # suggested on Advanced_Llama3_1_(3B)_GRPO_LoRA


# check lora code suggested on Advanced_Llama3_1_(3B)_GRPO_LoRA
from safetensors import safe_open
tensors = {}
with safe_open(f"{checkpoint_dir}/grpo_saved_lora_fix/adapter_model.safetensors", framework = "pt") as f:
    # Verify both A and B are non zero
    for key in f.keys():
        tensor = f.get_tensor(key)
        n_zeros = (tensor == 0).sum() / tensor.numel()
        assert(n_zeros.item() != tensor.numel())
print(f"OK") # It prints "OK", so the lora seems trained

# inference

text = tokenizer.apply_chat_template([
    {"role" : "system", "content" : SYSTEM_PROMPT},
    {
      "role" : "user",
      "content" : "Quante R ci sono nella parola ramarro?"
    },
], tokenize = False, add_generation_prompt = True)

#from vllm.lora.request import LoRARequest
from vllm import SamplingParams
sampling_params = SamplingParams(
    temperature = 0.9, # same as training
    top_p = 0.95,
    max_tokens = 8000,
    stop = ["</risposta>"]
)
output = model.fast_generate(
    text,
    sampling_params = sampling_params,
    # Each of them produce the same "vanilla" results
    lora_request = model.load_lora(f"{checkpoint_dir}/grpo_saved_lora_fix"),
    #lora_request = LoRARequest('lora', 1, lora_path='/content/checkpoint/checkpoint-1040')
)[0].outputs[0].text

print(output)

# push the merged model

user_id = "userd_id" # CHANGE ME
model_name = 'model_name'
print(user_id)
print(model_name)
print(f"Pushing to {user_id}/{model_name}")

# Just LoRA adapters
model.save_pretrained_merged(f"{model_name}-lora-only", tokenizer, save_method = "lora")
model.push_to_hub_merged(f"{user_id}/{model_name}-lora-only", tokenizer, save_method = "lora", token = os.getenv("HF_TOKEN"))

# Merge to 16bit
import os
model.save_pretrained_merged(model_name, tokenizer, save_method = "merged_16bit")
model.push_to_hub_merged(f"{user_id}/{model_name}", tokenizer, save_method = "merged_16bit", token = os.getenv("HF_TOKEN"))

@nazarenodefrancescomaize this problem can be easily solved by remove the len() as stated by @SpaceHunterInf

However, even if you solved this one, you will still face another problem that the saved lora are wrong.

Hey guys - apologies on the delay - specifically for GRPO - it’s fixed! I also made an advanced GRPO notebook which is actually much better than the original - it uses an A100 and does LoRA on Llama 3.2 3B, since it was faster.

NO NEED to retrain LoRA - if you saved it, then reload it! Unsloth inference is now fully fixed in the original GRPO and advanced notebook. (Please use advanced since I added many more reward functions, and it’s much better for convergence!

Advanced Colab notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_1_(3B)_GRPO_LoRA.ipynb

I also show the previous model without GRPO for the question "What is the sqrt of 101?"

Image

The trained GRPO on the same question - which definitely show it’s working!

Image

Using the following works without changing code - I fixed model.lora_lora

output = model.fast_generate(
    text,
    sampling_params = sampling_params,
    lora_request = model.load_lora("grpo_saved_lora"),
)[0].outputs[0].text

I also restarted the entire Colab, and just load the LoRA via the above (without training), and it worked correctly!

You can test if the LoRA is working by just counting the formatting:

no_trained_outputs = model.fast_generate(
    [text]*30,
    sampling_params = sampling_params,
    lora_request = None,
)
no_trained_outputs = [x.outputs[0].text for x in no_trained_outputs]
trained_outputs = model.fast_generate(
    [text]*30,
    sampling_params = sampling_params,
    lora_request = model.load_lora("grpo_saved_lora"),
)
trained_outputs = [x.outputs[0].text for x in trained_outputs]

symbols = [reasoning_start, reasoning_end, solution_start, solution_end]
print(sum(sum(x.count(y) == 1 for y in symbols) for x in no_trained_outputs))
print(sum(sum(x.count(y) == 1 for y in symbols) for x in trained_outputs))

You should get more good results as GRPO follows the formatting correctly for the trained outputs - if you don’t see any changes, then GRPO failed.

My training loss curves for the advanced notebook:

Image

Steps to get the latest updates:

  1. Update Unsloth via versioning to the latest stable:
pip install --upgrade --force-reinstall \
"unsloth==2025.3.18" "unsloth-zoo==2025.3.16"
  1. You do NOT need to retrain the LoRA - simply load it. I also added a LoRA error check to the advanced Colab to check if it’s actually trained!

  2. Gemma 3 does not use vLLM for now - this does NOT affect it.

Tagging everyone since I’m assuming this was important! Again apologies on the delay - there were tonnes of new models and stuff, so hopefully this helps! @kalachx @SpaceHunterInf @JangDongHyeon @caiqizh @accemlcc @llzllx70 @FloLey @shimmyshimmer @Alpacolypse @nazarenodefrancescomaize @YaqiWangCV @zzz1YAO @JangDongHyeon11

You are not alone, it is related to #1521

So merging LoRA to the base model is an issue, and even for those who successfully merge and try to inference the merged model, they still face an issue where performance deteriorates ???

🚨 Error loading merged float16 model from Hugging Face

I fine-tuned Qwen2-vl-7b for tomato leaf disease detection using Unsloth. After fine-tuning, I uploaded the merged model in float16 to Hugging Face using Lightning.AI on a T4 GPU (Colab kept crashing).

model.push_to_hub_merged("havoic206/Qwen2-vl-7b-TomatoLeafDoctor", tokenizer, token=HF_TOKEN)

Now, when I try to load the model:

model, tokenizer = FastVisionModel.from_pretrained(
    "havoic206/Qwen2-vl-7b-TomatoLeafDoctor",
    load_in_4bit=True,
)

On Lightning AI, I get this error:

FileNotFoundError: No such file or directory: "havoic206/Qwen2-vl-7b-TomatoLeafDoctor/model-00001-of-00004.safetensors"

On Colab, I get this:

OSError: havoic206/Qwen2-vl-7b-TomatoLeafDoctor does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.

The files in my repo look like this:

Image

I successfully load the LoRA weights from my Hugging Face repository, and the model returns responses that align with the training performed. Only issue is loading the merged model.

So we investigated the issue and were able to reproduce the issue (results after merging in 16-bit are same as before training).

However, we checked that _merge_lora is working as expected and that the base_layer weights before / after merging are indeed different by roughly the magnitude of B @ A (since the scale factor is 1 per the default Qwen-GRPO notebook).

We will need to conduct a deeper investigation into this thank you.

Do you guys know when this issue started firstly occuring?

I loaded in 4-bit and also have the same issued

I think it would be better if the tests have the same prompt before and after load lora. As it can shows that the improve come from lora but not the system_prompt

@Alpacolypse @FloLey @JangDongHyeon @JangDongHyeon11 @Macharia-Maurice @Mtrya @Serzhanov @SpaceHunterInf @YaqiWangCV @accemlcc @adamtcdev @akbarsigit @allen20200111 @caiqizh @daegonYu @gurveervirk @ikirtanee @jeromeku @jotaefe0 @kalachx @llzllx70 @metalwhale @nazarenodefrancescomaize @ngohoangnam28101995 @shed-e @void-mckenzie @youbin2014 @zzz1YAO

A fix has now been merged into pypi for both vision and language models and extensively tested. reference: https://github.com/unslothai/unsloth-zoo/pull/135 https://github.com/unslothai/unsloth-zoo/pull/158 https://github.com/unslothai/unsloth/pull/2673 Merging and saving should work now.

Best Practice is save to 16bits (which is the default operation if you do not specify anything), and load in 4 bits if you have to. For production systems, you might wanna consider post training quantization methods with calibration. These take longer but can also be more reliable. Saving and merging to 4bits might degrade performance of the model.

To access this recent fix, and if you’re running this in a local python environment, please make sure to upgrade your python environment using

pip install --upgrade unsloth_zoo
pip install --upgrade unsloth

If you’re in colab, the notebooks install cell will also be updated shortly but should be something like

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
    !pip install unsloth
else:
    # Do this only in Colab notebooks! Otherwise use pip install unsloth
    !pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton transformers cut_cross_entropy unsloth_zoo
    !pip install sentencepiece protobuf "datasets>=3.4.1" huggingface_hub hf_transfer
    !pip install --no-deps unsloth

as we’re no longer pinning transformers or trl.

Also note that whenever you need to merge while saving or while pushing to hub, make sure to use

save_pretrained_merged()

and

push_to_hub_merged()

If you are not merging at all, use

save_pretrained

and

push_to_hub

instead.

This has been closed for now. Feel free to comment if you face any problems. Happy finetuning.

Hi guys we have likely resolved the issue. Please update unsloth and let us know if it works! Thank you

change the following code:

–> 119 if len(lora_request.lora_tensors) is not None: to: if lora_request.lora_tensors is not None:

It works for me, ths

Trying to help debug the issue as relates to GRPO + Qwen

TLDR: the merged model seems to be working but there are a number of non-deterministic effects that I haven’t been able to pin down between sampling params, cudagraphs, and vllm.

This is based on:

unsloth                           2025.3.14     
unsloth_zoo                       2025.3.12 
vllm                              0.7.3

Did 2 training runs for the Qwen GRPO notebook, which I refactored into a script:

  • Using vllm for rollouts with cudagraphs enabled (default)
  • Using vllm for rollout without cudagraphs (had to turn it off by manually adding enforce_eager when instantiating vllm within unsloth.models.llama)

After training and merging models to 16bit, I ran inference across these scenarios:

  • Merged Models vs Base Model + trained adapter
  • Native HF / unsloth generate vs vllm as inference engine
  • If using vllm as inference engine, run with and without cudagraphs

For each of these, I ran with temperature 0.0 (greedy) and 0.8 (same as during training).

Observations:

  • Results that were most consistent with the results at the end of training
    • native hf / unsloth generate and greedy decoding (both merged and base + trained lora)
    • vllm generation for merged models under both training scenarios with greedy decoding – results were essentially same as base model if temperature same as during training (0.8)
  • For the base model + trained lora
    • For model trained WITHOUT vllm cudagraphs, the results were essentially same as untrained models under ALL vllm generation scenarios (with and without cudagraphs) EXCEPT for when generating with temperature 0.8 AND cudagraphs. This is truly confusing as model generation without cudagraphs using the model trained without cudagraphs should be reproducible under this scenario.
    • For the model trained WITH vllm cudagraphs, generation yields similar results with result at end of training when using cudagraphs during generation and with temperature same as during training – this makes sense. If I set temperature to 0.0 during generation and generation with / without cudagraphs, the results are similar to untrained model.

@FloLey - you mentioned in #1763 that you were observing that merging the model results in performance regression to the original untrained model. This was for a non-GRPO run (i.e., independent of vllm for fast_inference)? Would be good to start from here to isolate effects without additional confounders of vllm +/- cudagraphs.

Hi guys we have likely resolved the issue. Please update unsloth and let us know if it works! Thank you

Currently, the issue is not only with GRPO, but also when training the model with LoRA and merging it, the model does not change.

Image

[rank0]: File “~/anaconda3/envs/Unsloth/lib/python3.10/site-packages/unsloth_zoo/vllm_utils.py”, line 1202, in load_lora [rank0]: raise OSError(f"Unsloth: LoRA filepath = {save_directory} does not exist!")

[rank0]: OSError: Unsloth: LoRA filepath = grpo_saved_lora does not exist!

I would say it is not only big bug but a complete deal breaker for the project. Such issues show complete lack of testing being done before making releases. If something as mundane as saving model doesn’t work then what certainty can I have that other features work? For example whole models might have bugs because each model family needs specific support and why there are always patches for new models - since no one tests this stuff how do I know some older model family still works? Change something, do wrong tests (read: see that training starts and not that it actually works) or no tests at all and then user sits there waiting for their model to cook to then end up disappointed. As new user I experience only frustration and zero results. Did thousands of GPRO training steps and nothing. Model behaves identical to untrained model.

Yeah… it seems like it’s the exact same issue. I am using the lastest release though. Have you ever find any workaround for this except using the resume_training trick? How do othe people saving trained model now? I thought this should be some urgent bug that needs to be fixed.

So everyone else just finetune the entire model instead of using LoRA???