unsloth: Inconsistent Inference Results After Saving and Loading LoRA Model
I was following the official Google Colab notebook to train my reasoning LoRA model on Paperspace. I was able to run the notebook without any issues.
However, when trying to save the trained model, I encountered some unexpected behavior. I have attempted multiple methods to save the trained model/LoRA, but none of them produce the same chat response after loading (I’m using temperature = 0.0).
Additionally, if I save the LoRA using model.save_lora("grpo_saved_lora"), After restarting and kernel and reload the model, I get the exact same inference response whether I use lora_request = None or load it with model.load_lora("grpo_saved_lora").
Issue Details
I consistently use SYSTEM_PROMPT to construct the prompt, and my test question is:
“How many r’s are in strawberry?”
Responses Right After Training
- Without LoRA:
<reasoning> To determine how many times the letter 'r' appears in the word "strawberry", I will go through the word character by character and count each occurrence of 'r'. </reasoning> <answer> The letter 'r' appears 3 times in the word "strawberry". </answer> - With LoRA:
<reasoning>\nTo determine the number of \'r\'s in the word "strawberry," we need to count each occurrence of the letter \'r\' in the word. Let\'s examine the word character by character:\n\n1. s\n2. t\n3. r\n4. a\n5. w\n6. b\n7. y\n8. r\n9. r\n\nWe can see that the letter \'r\' appears twice in the word "strawberry."\n</reasoning>\n<answer>\n2\n</answer>
Responses After Saving and Reloading
Method 1: model.save_lora(“grpo_saved_lora”), Restart Kernel, Load with model.load_lora(“grpo_saved_lora”)
- With or Without LoRA:
'<reasoning>\nTo determine how many times the letter \'r\' appears in the word "strawberry", we can go through the word character by character and count each occurrence of \'r\'. \n\n- The first character is \'s\', not \'r\'.\n- The second character is \'t\', not \'r\'.\n- The third character is \'r\', this is the first occurrence of \'r\'.\n- The fourth character is \'w\', not \'r\'.\n- The fifth character is \'a\', not \'r\'.\n- The sixth character is \'r\', this is the second occurrence of \'r\'.\n- The seventh character is \'r\', this is the third occurrence of \'r\'.\n- The eighth character is \'b\', not \'r\'.\n- The ninth character is \'b\', not \'r\'.\n- The tenth character is \'e\', not \'r\'.\n\nBy going through the word, we find that the letter \'r\' appears 3 times.\n</reasoning>\n<answer>\n3\n</answer>'
Method 2: model.save_pretrained_merged("qwen-2.5-3b-lora-251", tokenizer, save_method="lora"), Load with FastLanguageModel.from_pretrained(...)
- Without LoRA: Same result as Method 1.
- With LoRA: Error - ‘Qwen2ForCausalLM’ object has no attribute ‘load_lora’.
Workaround
The only way I found to get consistent responses is to use the “Finetuning from Last Checkpoint” method:
- Load the model.
- Execute the same training code, but use
trainer.train(resume_from_checkpoint=True). Ensuremax_steps <= save_stepsto stop training immediately.
By doing this, the inference response matches what I got during the initial training.
I’m unsure if this is a bug or if I’m missing something in the saving/loading process. Could you please advise?
Thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 8
- Comments: 25 (2 by maintainers)
Hey all, I managed to make it reproducible. The only way I managed to load the adapter is via the trained hack described by @Xiaohan-Tian.
I trained using GRPO and a random reward with an absurd LR, in order to break the model and make the effect of loading the adapter more clearly visible. Please see attached notebook for more details 😃 (i had to rename it to .txt, as github doesn’t allow ipynb, sorry.)
try_to_load_model.ipynb
@Xiaohan-Tian Hey, did you figure out how to load the lora adapter?
@Xiaohan-Tian I am facing the same issue. Did you find the solution?
I’m able to load the model, but the inference results I’m getting now aren’t consistent with what I saw immediately after training (w/LoRA). I’ll investigate further tomorrow.
This fix has not addressed the issue.
I have the same issue, I think this should be an urgent bug to fix.