unsloth: Applying LoRA Doesn't Change Model Output
Greetings,
I am extremely confused why my model generates consistent result w/o my rank=64 LoRA. What’s even more confusing is, the LoRA works in my notebook after training. But whenn I start fresh, loading the LoRA again, it gives the output of the original model.
model_name=="Qwen/Qwen2.5-1.5B-Instruct",
vllm==0.7.3
trl==0.15.2
unsloth==2025.2.14
unsloth_zoo==2025.2.7
So I followed the tutorial notebook, trained the model using GRPO trainer. Then within the notebook, I model.save_lora("grpo_saved_lora"), then I do this using vllm with greedy decoding:
sampling_params = SamplingParams(
max_tokens=32, # Maximum number of tokens to generate
temperature=0, # Temperature of 0 means no randomness (deterministic)
top_p=1.0, # Consider all tokens in the distribution
top_k=1 # Only select the most likely token at each step
)
outputs = model.fast_generate(
test_split,
sampling_params = sampling_params,
lora_request = model.load_lora("grpo_saved_lora"),
)
These are all fine, I have different set of output compare with before training. But then, when I try to load the model lora with a separate script. The output is suddenly consistent with the original model, even I loaded the lora in this way.
cfg = load_config(args.config)
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=cfg.model_name,
max_seq_length=cfg.max_seq_length,
load_in_4bit=cfg.load_in_4bit,
fast_inference=cfg.fast_inference,
max_lora_rank=cfg.lora_rank,
gpu_memory_utilization=cfg.gpu_memory_utilization,
)
lora = None
if args.lora_path is not None:
model = FastLanguageModel.get_peft_model(
model,
r=cfg.lora_rank,
target_modules=cfg.target_modules,
lora_alpha=cfg.lora_rank,
use_gradient_checkpointing="unsloth",
random_state=3407,
)
lora = model.load_lora(args.lora_path)
print('using lora ' + args.lora_path)
_, _, test_dataset = get_dataset("ecqa")
sampling_params = SamplingParams( # Greedy decoding
max_tokens=32, # Maximum number of tokens to generate
temperature=0, # Temperature of 0 means no randomness (deterministic)
top_p=1.0, # Consider all tokens in the distribution
top_k=1 # Only select the most likely token at each step
)
vllm_prompts = [x['prompt'] for x in test_dataset]
outputs = model.fast_generate(
vllm_prompts,
sampling_params = sampling_params,
lora_request = lora
)
Could you please help me? I am really confused.
Thank you very much.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 51 (4 by maintainers)
Hi apologies everyone - let me investigate and I’ll see what I can do to fix the issue!
Yes. I have spent hundreds of dollars on A100 in order to finetune with GRPO Phi-4 and now that model is stuck on the virtual runtime because I cannot merge the model and push it to HF, and neither pushing only the lora works, because applying that lora gives only vanilla outputs.
I also tried retraining with latest versions of unsloth using smaller models (phi-4-mini and gemma-3), also using their own colabs without touching anything. Nothing. For now, it is a waste of money, I’m sorry to say that.
I think the problem is not fixed yet. I reloaded an already done training as suggested but the saved lora and the hf pushed ones have the same behaviour of the original model. I’m on “unsloth==2025.3.18” “unsloth-zoo==2025.3.16”
The wird thing is that with the previous version of unlosth (2025.3.13) the saved lora worked, the merged and pushed model not.
The training code:
@nazarenodefrancescomaize this problem can be easily solved by remove the len() as stated by @SpaceHunterInf
However, even if you solved this one, you will still face another problem that the saved lora are wrong.
You are not alone, it is related to https://github.com/unslothai/unsloth/issues/1521
Hey guys - apologies on the delay - specifically for GRPO - it’s fixed! I also made an advanced GRPO notebook which is actually much better than the original - it uses an A100 and does LoRA on Llama 3.2 3B, since it was faster.
NO NEED to retrain LoRA - if you saved it, then reload it! Unsloth inference is now fully fixed in the original GRPO and advanced notebook. (Please use advanced since I added many more reward functions, and it’s much better for convergence!
Advanced Colab notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_1_(3B)_GRPO_LoRA.ipynb
I also show the previous model without GRPO for the question
"What is the sqrt of 101?"The trained GRPO on the same question - which definitely show it’s working!
Using the following works without changing code - I fixed
model.lora_loraI also restarted the entire Colab, and just load the LoRA via the above (without training), and it worked correctly!
You can test if the LoRA is working by just counting the formatting:
You should get more good results as GRPO follows the formatting correctly for the trained outputs - if you don’t see any changes, then GRPO failed.
My training loss curves for the advanced notebook:
Steps to get the latest updates:
You do NOT need to retrain the LoRA - simply load it. I also added a LoRA error check to the advanced Colab to check if it’s actually trained!
Tagging everyone since I’m assuming this was important! Again apologies on the delay - there were tonnes of new models and stuff, so hopefully this helps! @kalachx @SpaceHunterInf @JangDongHyeon @caiqizh @accemlcc @llzllx70 @FloLey @shimmyshimmer @Alpacolypse @nazarenodefrancescomaize @YaqiWangCV @zzz1YAO @JangDongHyeon11
So merging LoRA to the base model is an issue, and even for those who successfully merge and try to inference the merged model, they still face an issue where performance deteriorates ???
🚨 Error loading merged float16 model from Hugging Face
I fine-tuned
Qwen2-vl-7bfor tomato leaf disease detection using Unsloth. After fine-tuning, I uploaded the merged model in float16 to Hugging Face using Lightning.AI on a T4 GPU (Colab kept crashing).Now, when I try to load the model:
On Lightning AI, I get this error:
On Colab, I get this:
The files in my repo look like this:
I successfully load the LoRA weights from my Hugging Face repository, and the model returns responses that align with the training performed. Only issue is loading the merged model.
I loaded in 4-bit and also have the same issued
I think it would be better if the tests have the same prompt before and after load lora. As it can shows that the improve come from lora but not the system_prompt
@Alpacolypse @FloLey @JangDongHyeon @JangDongHyeon11 @Macharia-Maurice @Mtrya @Serzhanov @SpaceHunterInf @YaqiWangCV @accemlcc @adamtcdev @akbarsigit @allen20200111 @caiqizh @daegonYu @gurveervirk @ikirtanee @jeromeku @jotaefe0 @kalachx @llzllx70 @metalwhale @nazarenodefrancescomaize @ngohoangnam28101995 @shed-e @void-mckenzie @youbin2014 @zzz1YAO
A fix has now been merged into pypi for both vision and language models and extensively tested. reference: https://github.com/unslothai/unsloth-zoo/pull/135 https://github.com/unslothai/unsloth-zoo/pull/158 https://github.com/unslothai/unsloth/pull/2673 Merging and saving should work now.
Best Practice is save to 16bits (which is the default operation if you do not specify anything), and load in 4 bits if you have to. For production systems, you might wanna consider post training quantization methods with calibration. These take longer but can also be more reliable. Saving and merging to 4bits might degrade performance of the model.
To access this recent fix, and if you’re running this in a local python environment, please make sure to upgrade your python environment using
If you’re in colab, the notebooks install cell will also be updated shortly but should be something like
as we’re no longer pinning transformers or trl.
Also note that whenever you need to merge while saving or while pushing to hub, make sure to use
and
If you are not merging at all, use
and
instead.
This has been closed for now. Feel free to comment if you face any problems. Happy finetuning.
change the following code:
–> 119 if len(lora_request.lora_tensors) is not None: to: if lora_request.lora_tensors is not None:
It works for me, ths
Trying to help debug the issue as relates to
GRPO + QwenTLDR: the merged model seems to be working but there are a number of non-deterministic effects that I haven’t been able to pin down between sampling params, cudagraphs, and vllm.
This is based on:
Did 2 training runs for the Qwen GRPO notebook, which I refactored into a script:
vllmfor rollouts withcudagraphsenabled (default)vllmfor rollout withoutcudagraphs(had to turn it off by manually addingenforce_eagerwhen instantiatingvllmwithinunsloth.models.llama)After training and merging models to 16bit, I ran inference across these scenarios:
generatevsvllmas inference enginevllmas inference engine, run with and withoutcudagraphsFor each of these, I ran with
temperature0.0 (greedy) and 0.8 (same as during training).Observations:
generateand greedy decoding (bothmergedandbase + trained lora)vllmgeneration for merged models under both training scenarios with greedy decoding – results were essentially same as base model if temperature same as during training (0.8)vllmcudagraphs, the results were essentially same as untrained models under ALLvllmgeneration scenarios (with and withoutcudagraphs) EXCEPT for when generating withtemperature 0.8ANDcudagraphs. This is truly confusing as model generation withoutcudagraphsusing the model trained withoutcudagraphsshould be reproducible under this scenario.vllmcudagraphs, generation yields similar results with result at end of training when usingcudagraphsduring generation and with temperature same as during training – this makes sense. If I settemperatureto0.0during generation and generation with / withoutcudagraphs, the results are similar to untrained model.@FloLey - you mentioned in #1763 that you were observing that merging the model results in performance regression to the original untrained model. This was for a non-GRPO run (i.e., independent of
vllmforfast_inference)? Would be good to start from here to isolate effects without additional confounders ofvllm+/-cudagraphs.Hi guys we have likely resolved the issue. Please update unsloth and let us know if it works! Thank you
Currently, the issue is not only with GRPO, but also when training the model with LoRA and merging it, the model does not change.
[rank0]: File “~/anaconda3/envs/Unsloth/lib/python3.10/site-packages/unsloth_zoo/vllm_utils.py”, line 1202, in load_lora [rank0]: raise OSError(f"Unsloth: LoRA filepath = {save_directory} does not exist!")
[rank0]: OSError: Unsloth: LoRA filepath = grpo_saved_lora does not exist!
I would say it is not only big bug but a complete deal breaker for the project. Such issues show complete lack of testing being done before making releases. If something as mundane as saving model doesn’t work then what certainty can I have that other features work? For example whole models might have bugs because each model family needs specific support and why there are always patches for new models - since no one tests this stuff how do I know some older model family still works? Change something, do wrong tests (read: see that training starts and not that it actually works) or no tests at all and then user sits there waiting for their model to cook to then end up disappointed. As new user I experience only frustration and zero results. Did thousands of GPRO training steps and nothing. Model behaves identical to untrained model.
Yeah… it seems like it’s the exact same issue. I am using the lastest release though. Have you ever find any workaround for this except using the resume_training trick? How do othe people saving trained model now? I thought this should be some urgent bug that needs to be fixed.
So everyone else just finetune the entire model instead of using LoRA???
I think this is the same issue as all those: https://github.com/unslothai/unsloth/issues/1670 https://github.com/unslothai/unsloth/issues/1763 https://github.com/unslothai/unsloth/issues/1877 https://github.com/unslothai/unsloth/issues/1929 https://github.com/unslothai/unsloth/issues/1971