unsloth: GRPOTrainer crashes with unsloth
I am trying to run GRPOTrainer with unsloth but it crashes. How to fix this? unsloth 2025.2.4 unsloth 2025.2.3 transformers 4.47.1 torch 2.5.1 trl 0.14.0
This is the relevant code:
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = base_model,
max_seq_length = 2048,
attn_implementation="flash_attention_2",
dtype = torch.bfloat16,
load_in_4bit = True,
)
training_args = GRPOConfig(
output_dir=output_dir,
learning_rate=5e-6,
adam_beta1 = 0.9,
adam_beta2 = 0.99,
weight_decay = 0.05,
bf16=True,
warmup_ratio = 0.1,
lr_scheduler_type='cosine',
logging_steps=1,
per_device_train_batch_size=1,
gradient_accumulation_steps=2,
num_generations=8,
max_prompt_length=256,
max_completion_length=786,
num_train_epochs=1,
save_steps=steps_num,
save_total_limit=2,
max_grad_norm=0.1,
report_to="none",
log_on_each_node=False,
)
model = FastLanguageModel.get_peft_model(
model,
r = 32, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 32,
lora_dropout = 0, # Currently only supports dropout = 0
bias = "none", # Currently only supports bias = "none"
use_gradient_checkpointing = "unsloth",
random_state = 3407,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)
trainer = GRPOTrainer(
model=model,
processing_class=tokenizer,
reward_funcs=reward_func,
args=training_args,
train_dataset=dataset,
)
trainer.train()
This is the message when it crashes:
Traceback (most recent call last):
File "/home/user/ft/grpo.py", line 184, in <module>
trainer.train()
File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 2164, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "<string>", line 382, in _fast_inner_training_loop
File "<string>", line 31, in _unsloth_training_step
File "/home/user/anaconda3/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py", line 422, in compute_loss
prompt_completion_ids = unwrapped_model.generate(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/peft/peft_model.py", line 1838, in generate
outputs = self.base_model.generate(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/generation/utils.py", line 2252, in generate
result = self._sample(
^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/generation/utils.py", line 3251, in _sample
outputs = self(**model_inputs, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/llama.py", line 1025, in _CausalLM_fast_forward
outputs = fast_forward_inference(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/gemma2.py", line 397, in Gemma2Model_fast_forward_inference
seq_len = past_key_values[0][0].shape[-2]
~~~~~~~~~~~~~~~^^^
File "<string>", line 10, in __cache_utils_getitem__
RuntimeError: Unsloth: You must call `FastLanguageModel.for_inference(model)` before doing inference for Unsloth models.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 37 (5 by maintainers)
I can confirm that unsloth 2025.3.19 works with gemma 2 and 3 by turning off vllm. However, it crashes on both when vllm is on. Since unsloth is too slow to be useful without vllm, so I suppose I will keep this issue open.
The correct fix is to
pip install diffusers.Fixed myself. Forgot to add fast_inference=True to FastLanguageModel.from_pretained
However, it core dumped on me. Is this an OOM error?
I find that if you manually change ~/anaconda3/lib/python3.12/site-packages/trl/trainer/alignprop_trainer.py from
from …models import DDPOStableDiffusionPipeline
to
from …models.modeling_sd_base import DDPOStableDiffusionPipeline
Then it can run.
Guys I got it
This works, not sure you guys installed
diffusersas wellIt should work now! Please update Unsloth via
pip install --upgrade --force-reinstall --no-cache-dir unsloth_zoo unsloththen before your script, do: