unsloth: AttributeError: 'HybridCache' object has no attribute 'float'—Gemma 3 training fails with BF16 precision on RTX3090 (Ampere) GPUs
I have an NVidia RTX3090 GPU on Linux. It appears that Unsloth has issues with BF16 (and FP16) training on the recently released Gemma-3 and this seems a problem related with the GPU.
- A fresh Conda environment was used to troubleshoot the issue, installed according to the documentation on the Unsloth website.
- The latest Unsloth was also installed, the main libraries from
pipbeing:
transformers 4.50.0.dev0
unsloth 2025.3.14
unsloth_zoo 2025.3.12
xformers 0.0.29.post3
In summary, with bf16=True training fails with an AttributeError: 'HybridCache' object has no attribute 'float' error. The traceback is as follows:
Traceback (most recent call last):
File "/home/user/Datasets/negative/mge_train.py", line 218, in <module>
trainer.train(resume_from_checkpoint = False)
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 2251, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "<string>", line 221, in _fast_inner_training_loop
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 3053, in _evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 4135, in evaluate
output = eval_loop(
^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 4329, in evaluation_loop
losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 4545, in prediction_step
loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Datasets/negative/unsloth_compiled_cache/UnslothSFTTrainer.py", line 749, in compute_loss
outputs = super().compute_loss(
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/_utils.py", line 1025, in _unsloth_pre_compute_loss
outputs = self._old_compute_loss(model, inputs, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 3789, in compute_loss
outputs = model(**inputs)
^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 819, in forward
return model_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 807, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 786, in convert_to_fp32
return recursively_apply(_convert_to_fp32, tensor, test_type=_is_fp16_bf16_tensor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 118, in recursively_apply
{
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 119, in <dictcomp>
k: recursively_apply(
^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 126, in recursively_apply
return func(data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/accelerate/utils/operations.py", line 778, in _convert_to_fp32
return tensor.float()
^^^^^^^^^^^^
AttributeError: 'HybridCache' object has no attribute 'float'
While with fp16 = True this occurs (although I understand this is expected):
Traceback (most recent call last):
File "/home/anon/Datasets/negative/mge_train.py", line 102, in <module>
trainer = SFTTrainer(
^^^^^^^^^^^
File "/home/anon/bin/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/trainer.py", line 203, in new_init
original_init(self, *args, **kwargs)
File "/home/anon/Datasets/negative/unsloth_compiled_cache/UnslothSFTTrainer.py", line 923, in __init__
if not force_float32 and (not float16 and use_fp16): raise TypeError('Unsloth: Model is in bfloat16 precision but you want to use float16 precision. Set fp16 to `False` and bf16 to `True`')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Unsloth: Model is in bfloat16 precision but you want to use float16 precision. Set fp16 to `False` and bf16 to `True`
As a workaround, training works after configuring in the SFTTrainer configuration:
fp16=False,
bf16=False,
Current nvidia-smi output:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04 Driver Version: 570.124.04 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:01:00.0 Off | N/A |
| 0% 39C P8 19W / 250W | 22789MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 9110 G /usr/bin/Xorg.bin 4MiB |
| 0 N/A N/A 1454055 C ../koboldcpp-linux-x64-cuda1210 22766MiB |
+-----------------------------------------------------------------------------------------+
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 25
Hi @BugReporterZ and everyone following this issue,
Let me explain what’s happening here:
Regarding fp16=True error:
Gemma 3 was trained using bf16 precision. When you set
fp16=True, or when your GPU does not support bf16, Unsloth automatically upcasts to 32-bit to avoid potential instability from using fp16. This is intentional behavior, not a bug. The error message you’re seeing is actually telling you this - you shouldn’t use fp16 with Gemma 3 models.Regarding bf16=True error:
This is caused by an interaction between the transformers and accelerate libraries. It is not an unsloth bug. The root issue is:
.float()method, causing the AttributeErrorThe accelerate team is actively discussing the best way to fix this (see accelerate PR 3451 and related issues).
Workarounds:
use_cache = Falseduring training . check the config.json model file to see where the use_cache key is located at.Truefor inference, as the model needs KV caching for generationI’ll look into whether to add a temporary monkey patch in Unsloth to handle this case more gracefully until the accelerate library implements a proper fix. If not, the workaround will have to do until accelerate commit that PR.
Hi folks! The issue is on the
transformers-side, and this PR fixes it 🤗TL;DR having classes with tensor-like attributes may cause other libraries to treat them as tensors
Understandable. Follow my instructions above-
@BugReporterZ @Serzhanov @BM-K @flofiz @Arnob-MagierAi @feliciaebbot @aliswh @legekka @Kuangdd01 @machlovi @IridiumMaster @phiwi
What @rolandtannous said:
Sorry if I’m being a little dense here, could you make an illustration? Or am I just setting: model.config.text_config.use_cache = False
So I ran : sudo find / -name config.json 2>/dev/null
Then edited the config.json that unsloth downloaded. But now I get this message: You have set
use_cachetoFalse, but cache_implementation is set to hybrid. cache_implementation will have no effect. Running it to see if it works. Yes, that worked. Fair enough.Hi guys the issue is on transformers side should be fixed now thanks to the HF team! Going to close this thread for now. PR fix: https://github.com/huggingface/transformers/pull/37044
As gante said: TL;DR having classes with tensor-like attributes may cause other libraries to treat them as tensors
CC: @BugReporterZ @Serzhanov @BM-K @flofiz @Arnob-MagierAi @feliciaebbot @aliswh @legekka @ccs96307 @phiwi @Kuangdd01 @machlovi @IridiumMaster @iremeyiokur
Hi, I had the same issue, my problem solved with this PR without
use_cache=false.This fixed it for me!
I found that larger Gemma 3 models have different model configs than regular text based LLMs, since they are multimodal. To disable the hybrid cache, instead of
trainer.model.config.use_cache = Falseyou should initialize a model config whereconfig.text_config.use_cacheset toFalse.You can check the model config.json on huggingface to verify the correct path.
@aliswh trainer.model.config.use_cache = False