unsloth: AttributeError: 'HybridCache' object has no attribute 'float'—Gemma 3 training fails with BF16 precision on RTX3090 (Ampere) GPUs

I have an NVidia RTX3090 GPU on Linux. It appears that Unsloth has issues with BF16 (and FP16) training on the recently released Gemma-3 and this seems a problem related with the GPU.

  • A fresh Conda environment was used to troubleshoot the issue, installed according to the documentation on the Unsloth website.
  • The latest Unsloth was also installed, the main libraries from pip being:
transformers             4.50.0.dev0
unsloth                  2025.3.14
unsloth_zoo              2025.3.12
xformers                 0.0.29.post3

In summary, with bf16=True training fails with an AttributeError: 'HybridCache' object has no attribute 'float' error. The traceback is as follows:

Traceback (most recent call last):
  File "/home/user/Datasets/negative/mge_train.py", line 218, in <module>
    trainer.train(resume_from_checkpoint = False)
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 2251, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 221, in _fast_inner_training_loop
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 3053, in _evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 4135, in evaluate
    output = eval_loop(
             ^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 4329, in evaluation_loop
    losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 4545, in prediction_step
    loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/Datasets/negative/unsloth_compiled_cache/UnslothSFTTrainer.py", line 749, in compute_loss
    outputs = super().compute_loss(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/_utils.py", line 1025, in _unsloth_pre_compute_loss
    outputs = self._old_compute_loss(model, inputs, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 3789, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 786, in convert_to_fp32
    return recursively_apply(_convert_to_fp32, tensor, test_type=_is_fp16_bf16_tensor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 118, in recursively_apply
    {
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 119, in <dictcomp>
    k: recursively_apply(
       ^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 126, in recursively_apply
    return func(data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 778, in _convert_to_fp32
    return tensor.float()
           ^^^^^^^^^^^^
AttributeError: 'HybridCache' object has no attribute 'float'

While with fp16 = True this occurs (although I understand this is expected):

Traceback (most recent call last):
  File "/home/anon/Datasets/negative/mge_train.py", line 102, in <module>
    trainer = SFTTrainer(
              ^^^^^^^^^^^
  File "/home/anon/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/trainer.py", line 203, in new_init
    original_init(self, *args, **kwargs)
  File "/home/anon/Datasets/negative/unsloth_compiled_cache/UnslothSFTTrainer.py", line 923, in __init__
    if not force_float32 and (not float16 and use_fp16): raise TypeError('Unsloth: Model is in bfloat16 precision but you want to use float16 precision. Set fp16 to `False` and bf16 to `True`')
                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Unsloth: Model is in bfloat16 precision but you want to use float16 precision. Set fp16 to `False` and bf16 to `True`

As a workaround, training works after configuring in the SFTTrainer configuration:

fp16=False,
bf16=False,

Current nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04             Driver Version: 570.124.04     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:01:00.0 Off |                  N/A |
|  0%   39C    P8             19W /  250W |   22789MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            9110      G   /usr/bin/Xorg.bin                         4MiB |
|    0   N/A  N/A         1454055      C   ../koboldcpp-linux-x64-cuda1210       22766MiB |
+-----------------------------------------------------------------------------------------+

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 25

Most upvoted comments

Hi @BugReporterZ and everyone following this issue,

Let me explain what’s happening here:

  1. Regarding fp16=True error:
    Gemma 3 was trained using bf16 precision. When you set fp16=True, or when your GPU does not support bf16, Unsloth automatically upcasts to 32-bit to avoid potential instability from using fp16. This is intentional behavior, not a bug. The error message you’re seeing is actually telling you this - you shouldn’t use fp16 with Gemma 3 models.

  2. Regarding bf16=True error:
    This is caused by an interaction between the transformers and accelerate libraries. It is not an unsloth bug. The root issue is:

    • Transformers now declares Cache objects (like HybridCache) as Tensor types
    • accelerate tries to recursively convert bf16 tensors to fp32 during evaluation
    • The Cache objects don’t actually have a .float() method, causing the AttributeError

The accelerate team is actively discussing the best way to fix this (see accelerate PR 3451 and related issues).

Workarounds:

  • You can temporarily set use_cache = False during training . check the config.json model file to see where the use_cache key is located at.
  • Remember to set it back to True for inference, as the model needs KV caching for generation

I’ll look into whether to add a temporary monkey patch in Unsloth to handle this case more gracefully until the accelerate library implements a proper fix. If not, the workaround will have to do until accelerate commit that PR.

Hi folks! The issue is on the transformers-side, and this PR fixes it 🤗

TL;DR having classes with tensor-like attributes may cause other libraries to treat them as tensors

Thank you for these information.

I’m still a bit stuck. In the special case of google/gemma-3-27b-it, I both set

model.config.text_config.use_cache = False
trainer.model.config.use_cache = False

but this still leads to the AttributeError during evaluation. Am I missing something?

I’m trying now by just skipping evaluation.

Understandable. Follow my instructions above-

  1. Run your notebook so that the Gemma model gets downloaded first
  2. Find where unsloth downloaded the model by running the terminal command sudo find / -name config.json 2>/dev/null which will list all possible locations
  3. Edit the use_cache in the config you find so that it is false.
  4. Continue with notebook operations and don’t set the value anywhere else

@BugReporterZ @Serzhanov @BM-K @flofiz @Arnob-MagierAi @feliciaebbot @aliswh @legekka @Kuangdd01 @machlovi @IridiumMaster @phiwi

What @rolandtannous said:

Hi @BugReporterZ and everyone following this issue,

Let me explain what’s happening here:

  1. Regarding fp16=True error: Gemma 3 was trained using bf16 precision. When you set fp16=True, or when your GPU does not support bf16, Unsloth automatically upcasts to 32-bit to avoid potential instability from using fp16. This is intentional behavior, not a bug. The error message you’re seeing is actually telling you this - you shouldn’t use fp16 with Gemma 3 models.

  2. Regarding bf16=True error: This is caused by an interaction between the transformers and accelerate libraries. It is not an unsloth bug. The root issue is:

    • Transformers now declares Cache objects (like HybridCache) as Tensor types
    • accelerate tries to recursively convert bf16 tensors to fp32 during evaluation
    • The Cache objects don’t actually have a .float() method, causing the AttributeError

The accelerate team is actively discussing the best way to fix this (see accelerate PR 3451 and related issues).

Workarounds:

  • You can temporarily set use_cache = False during training . check the config.json model file to see where the use_cache key is located at.
  • Remember to set it back to True for inference, as the model needs KV caching for generation

I’ll look into whether to add a temporary monkey patch in Unsloth to handle this case more gracefully until the accelerate library implements a proper fix. If not, the workaround will have to do until accelerate commit that PR.

I found that larger Gemma 3 models have different model configs than regular text based LLMs, since they are multimodal. To disable the hybrid cache, instead of trainer.model.config.use_cache = False you should initialize a model config where config.text_config.use_cache set to False. You can check the model config.json on huggingface to verify the correct path.

This fixed it for me!

Sorry if I’m being a little dense here, could you make an illustration? Or am I just setting: model.config.text_config.use_cache = False

So I ran : sudo find / -name config.json 2>/dev/null

Then edited the config.json that unsloth downloaded. But now I get this message: You have set use_cache to False, but cache_implementation is set to hybrid. cache_implementation will have no effect. Running it to see if it works. Yes, that worked. Fair enough.

Hi guys the issue is on transformers side should be fixed now thanks to the HF team! Going to close this thread for now. PR fix: https://github.com/huggingface/transformers/pull/37044

As gante said: TL;DR having classes with tensor-like attributes may cause other libraries to treat them as tensors

CC: @BugReporterZ @Serzhanov @BM-K @flofiz @Arnob-MagierAi @feliciaebbot @aliswh @legekka @ccs96307 @phiwi @Kuangdd01 @machlovi @IridiumMaster @iremeyiokur

Hi, I had the same issue, my problem solved with this PR without use_cache=false.

I found that larger Gemma 3 models have different model configs than regular text based LLMs, since they are multimodal. To disable the hybrid cache, instead of trainer.model.config.use_cache = False you should initialize a model config where config.text_config.use_cache set to False.

You can check the model config.json on huggingface to verify the correct path.

This fixed it for me!

I found that larger Gemma 3 models have different model configs than regular text based LLMs, since they are multimodal. To disable the hybrid cache, instead of trainer.model.config.use_cache = False you should initialize a model config where config.text_config.use_cache set to False.

You can check the model config.json on huggingface to verify the correct path.

@aliswh trainer.model.config.use_cache = False