unsloth: GRPOTrainer crashes with unsloth

I am trying to run GRPOTrainer with unsloth but it crashes. How to fix this? unsloth 2025.2.4 unsloth 2025.2.3 transformers 4.47.1 torch 2.5.1 trl 0.14.0

This is the relevant code:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = base_model, 
    max_seq_length = 2048,
    attn_implementation="flash_attention_2",
    dtype = torch.bfloat16,
    load_in_4bit = True,
)

training_args = GRPOConfig(
    output_dir=output_dir,
    learning_rate=5e-6,
    adam_beta1 = 0.9,
    adam_beta2 = 0.99,
    weight_decay = 0.05,
    bf16=True,
    warmup_ratio = 0.1,
    lr_scheduler_type='cosine',
    logging_steps=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=2,
    num_generations=8,
    max_prompt_length=256,
    max_completion_length=786,
    num_train_epochs=1,
    save_steps=steps_num,
    save_total_limit=2,
    max_grad_norm=0.1,
    report_to="none",
    log_on_each_node=False,
)
model = FastLanguageModel.get_peft_model(
    model,
    r = 32, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 32,
    lora_dropout = 0, # Currently only supports dropout = 0
    bias = "none",    # Currently only supports bias = "none"
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)
trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=reward_func,
    args=training_args,
    train_dataset=dataset,
)
trainer.train()

This is the message when it crashes:

Traceback (most recent call last):
  File "/home/user/ft/grpo.py", line 184, in <module>
    trainer.train()
  File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 2164, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 382, in _fast_inner_training_loop
  File "<string>", line 31, in _unsloth_training_step
  File "/home/user/anaconda3/lib/python3.12/site-packages/trl/trainer/grpo_trainer.py", line 422, in compute_loss
    prompt_completion_ids = unwrapped_model.generate(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/peft/peft_model.py", line 1838, in generate
    outputs = self.base_model.generate(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/generation/utils.py", line 2252, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/generation/utils.py", line 3251, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/llama.py", line 1025, in _CausalLM_fast_forward
    outputs = fast_forward_inference(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/gemma2.py", line 397, in Gemma2Model_fast_forward_inference
    seq_len = past_key_values[0][0].shape[-2]
              ~~~~~~~~~~~~~~~^^^
  File "<string>", line 10, in __cache_utils_getitem__
RuntimeError: Unsloth: You must call `FastLanguageModel.for_inference(model)` before doing inference for Unsloth models.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 37 (5 by maintainers)

Most upvoted comments

I can confirm that unsloth 2025.3.19 works with gemma 2 and 3 by turning off vllm. However, it crashes on both when vllm is on. Since unsloth is too slow to be useful without vllm, so I suppose I will keep this issue open.

It should work now! Please update Unsloth via pip install --upgrade --force-reinstall --no-cache-dir unsloth_zoo unsloth then before your script, do: from unsloth import FastLanguageModel, PatchFastRL PatchFastRL(“GRPO”, FastLanguageModel)

Im getting this error: RuntimeError: Failed to import trl.trainer.alignprop_trainer because of the following error (look up to see its traceback): cannot import name 'DDPOStableDiffusionPipeline' from 'trl.models' ([/home/elior/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/trl/models/__init__.py](http://localhost:8888/home/elior/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/trl/models/__init__.py)) Already tried to reinstall trl but still can’t run it.

I find that if you manually change ~/anaconda3/lib/python3.12/site-packages/trl/trainer/alignprop_trainer.py from

from …models import DDPOStableDiffusionPipeline

to

from …models.modeling_sd_base import DDPOStableDiffusionPipeline

Then it can run.

The correct fix is to pip install diffusers.

Thanks for the reply. I am trying to fine tune gemma-2-2b but it got this vllm error:

But after I get rid of “use_vllm=True”, it seems to be running.

Fixed myself. Forgot to add fast_inference=True to FastLanguageModel.from_pretained

However, it core dumped on me. Is this an OOM error?

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/user/ft/grpo.py", line 200, in <module>
[rank0]:     trainer.train()
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 2171, in train
[rank0]:     return inner_training_loop(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "<string>", line 382, in _fast_inner_training_loop
[rank0]:   File "<string>", line 25, in _unsloth_training_step
[rank0]:   File "/home/user/ft/unsloth_compiled_cache/GRPOTrainer.py", line 375, in _prepare_inputs
[rank0]:     ref_per_token_logps = self._get_per_token_logps(
[rank0]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/ft/unsloth_compiled_cache/GRPOTrainer.py", line 271, in _get_per_token_logps
[rank0]:     logits = model(
[rank0]:              ^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/accelerate/utils/operations.py", line 819, in forward
[rank0]:     return model_forward(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/accelerate/utils/operations.py", line 807, in __call__
[rank0]:     return convert_to_fp32(self.model_forward(*args, **kwargs))
[rank0]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/_compile.py", line 32, in inner
[rank0]:     return disable_fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/llama.py", line 1183, in PeftModelForCausalLM_fast_forward
[rank0]:     return self.base_model(
[rank0]:            ^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 197, in forward
[rank0]:     return self.model.forward(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/llama.py", line 1043, in _CausalLM_fast_forward
[rank0]:     outputs = self.model(
[rank0]:               ^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/llama.py", line 836, in LlamaModel_fast_forward
[rank0]:     hidden_states = Unsloth_Offloaded_Gradient_Checkpointer.apply(
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/autograd/function.py", line 575, in apply
[rank0]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 465, in decorate_fwd
[rank0]:     return fwd(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth_zoo/gradient_checkpointing.py", line 147, in forward
[rank0]:     output = forward_function(hidden_states, *args)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/models/gemma2.py", line 226, in Gemma2DecoderLayer_fast_forward
[rank0]:     hidden_states = self.mlp(hidden_states)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/kernels/fast_lora.py", line 192, in apply_lora_mlp_geglu_approx
[rank0]:     out = LoRA_MLP.apply(X,
[rank0]:           ^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/autograd/function.py", line 575, in apply
[rank0]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 465, in decorate_fwd
[rank0]:     return fwd(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/kernels/fast_lora.py", line 77, in forward
[rank0]:     h = _forward_function(e, g)
[rank0]:         ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/unsloth/kernels/geglu.py", line 138, in geglu_approx_forward_kernel
[rank0]:     _approx_forward_kernel[grid](gate, up, out, n_elements, BLOCK_SIZE = 1024,)
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda>
[rank0]:     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
[rank0]:                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run
[rank0]:     kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
[rank0]:     ^^^^^^^^^^
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/triton/compiler/compiler.py", line 381, in __getattribute__
[rank0]:     self._init_handles()
[rank0]:   File "/home/user/anaconda3/lib/python3.12/site-packages/triton/compiler/compiler.py", line 376, in _init_handles
[rank0]:     self.module, self.function, self.n_regs, self.n_spills = driver.active.utils.load_binary(
[rank0]:                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x71a8be76c446 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x71a8be7166e4 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x71a8bf0a5a18 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x10219ec (0x71a866a219ec in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x102a735 (0x71a866a2a735 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x5fc5b0 (0x71a8afdfc5b0 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x6f69f (0x71a8be74d69f in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x71a8be74637b in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x71a8be746529 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x8ca268 (0x71a8b00ca268 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2e0 (0x71a8b00ca5d0 in /home/user/anaconda3/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #43: <unknown function> + 0x2a1ca (0x71a8c322a1ca in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #44: __libc_start_main + 0x8b (0x71a8c322a28b in /usr/lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

It should work now! Please update Unsloth via pip install --upgrade --force-reinstall --no-cache-dir unsloth_zoo unsloth then before your script, do: from unsloth import FastLanguageModel, PatchFastRL PatchFastRL(“GRPO”, FastLanguageModel)

Im getting this error: RuntimeError: Failed to import trl.trainer.alignprop_trainer because of the following error (look up to see its traceback): cannot import name 'DDPOStableDiffusionPipeline' from 'trl.models' ([/home/elior/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/trl/models/__init__.py](http://localhost:8888/home/elior/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/trl/models/__init__.py))

Already tried to reinstall trl but still can’t run it.

I find that if you manually change ~/anaconda3/lib/python3.12/site-packages/trl/trainer/alignprop_trainer.py from

from …models import DDPOStableDiffusionPipeline

to

from …models.modeling_sd_base import DDPOStableDiffusionPipeline

Then it can run.

Guys I got it

%%capture
# Skip restarting message in Colab
import sys; modules = list(sys.modules.keys())
for x in modules: sys.modules.pop(x) if "PIL" in x or "google" in x else None

!pip install unsloth vllm
!pip install --upgrade pillow
# If you are running this notebook on local, you need to install `diffusers` too
!pip install diffusers
# Temporarily install a specific TRL nightly version
!pip install git+https://github.com/huggingface/trl.git@e95f9fb74a3c3647b86f251b7e230ec51c64b72b

This works, not sure you guys installed diffusers as well

It should work now! Please update Unsloth via pip install --upgrade --force-reinstall --no-cache-dir unsloth_zoo unsloth then before your script, do:

from unsloth import FastLanguageModel, PatchFastRL
PatchFastRL("GRPO", FastLanguageModel)