unsloth: Inconsistent Inference Results After Saving and Loading LoRA Model

I was following the official Google Colab notebook to train my reasoning LoRA model on Paperspace. I was able to run the notebook without any issues.

However, when trying to save the trained model, I encountered some unexpected behavior. I have attempted multiple methods to save the trained model/LoRA, but none of them produce the same chat response after loading (I’m using temperature = 0.0).

Additionally, if I save the LoRA using model.save_lora("grpo_saved_lora"), After restarting and kernel and reload the model, I get the exact same inference response whether I use lora_request = None or load it with model.load_lora("grpo_saved_lora").

Issue Details

I consistently use SYSTEM_PROMPT to construct the prompt, and my test question is:
“How many r’s are in strawberry?”

Responses Right After Training

  • Without LoRA: <reasoning> To determine how many times the letter 'r' appears in the word "strawberry", I will go through the word character by character and count each occurrence of 'r'. </reasoning> <answer> The letter 'r' appears 3 times in the word "strawberry". </answer>
  • With LoRA: <reasoning>\nTo determine the number of \'r\'s in the word "strawberry," we need to count each occurrence of the letter \'r\' in the word. Let\'s examine the word character by character:\n\n1. s\n2. t\n3. r\n4. a\n5. w\n6. b\n7. y\n8. r\n9. r\n\nWe can see that the letter \'r\' appears twice in the word "strawberry."\n</reasoning>\n<answer>\n2\n</answer>

Responses After Saving and Reloading

Method 1: model.save_lora(“grpo_saved_lora”), Restart Kernel, Load with model.load_lora(“grpo_saved_lora”)

  • With or Without LoRA: '<reasoning>\nTo determine how many times the letter \'r\' appears in the word "strawberry", we can go through the word character by character and count each occurrence of \'r\'. \n\n- The first character is \'s\', not \'r\'.\n- The second character is \'t\', not \'r\'.\n- The third character is \'r\', this is the first occurrence of \'r\'.\n- The fourth character is \'w\', not \'r\'.\n- The fifth character is \'a\', not \'r\'.\n- The sixth character is \'r\', this is the second occurrence of \'r\'.\n- The seventh character is \'r\', this is the third occurrence of \'r\'.\n- The eighth character is \'b\', not \'r\'.\n- The ninth character is \'b\', not \'r\'.\n- The tenth character is \'e\', not \'r\'.\n\nBy going through the word, we find that the letter \'r\' appears 3 times.\n</reasoning>\n<answer>\n3\n</answer>'

Method 2: model.save_pretrained_merged("qwen-2.5-3b-lora-251", tokenizer, save_method="lora"), Load with FastLanguageModel.from_pretrained(...)

  • Without LoRA: Same result as Method 1.
  • With LoRA: Error - ‘Qwen2ForCausalLM’ object has no attribute ‘load_lora’.

Workaround

The only way I found to get consistent responses is to use the “Finetuning from Last Checkpoint” method:

  1. Load the model.
  2. Execute the same training code, but use trainer.train(resume_from_checkpoint=True). Ensure max_steps <= save_steps to stop training immediately.

By doing this, the inference response matches what I got during the initial training.


I’m unsure if this is a bug or if I’m missing something in the saving/loading process. Could you please advise?

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 8
  • Comments: 25 (2 by maintainers)

Most upvoted comments

Hey all, I managed to make it reproducible. The only way I managed to load the adapter is via the trained hack described by @Xiaohan-Tian.

I trained using GRPO and a random reward with an absurd LR, in order to break the model and make the effect of loading the adapter more clearly visible. Please see attached notebook for more details 😃 (i had to rename it to .txt, as github doesn’t allow ipynb, sorry.)

try_to_load_model.ipynb

@Xiaohan-Tian Hey, did you figure out how to load the lora adapter?

@Xiaohan-Tian I am facing the same issue. Did you find the solution?

I’m able to load the model, but the inference results I’m getting now aren’t consistent with what I saw immediately after training (w/LoRA). I’ll investigate further tomorrow.

This fix has not addressed the issue.

I have the same issue, I think this should be an urgent bug to fix.