unsloth: AttributeError: 'NoneType' object has no attribute 'attn_bias'
I got an error of xformers:
Traceback (most recent call last):
File "/data1/xuanyang/kooshot_cate_pred_llm_sft/sft.py", line 131, in <module>
trainer_stats = trainer.train()
^^^^^^^^^^^^^^^
File "<string>", line 157, in train
File "<string>", line 382, in _fast_inner_training_loop
File "<string>", line 31, in _unsloth_training_step
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/unsloth/models/_utils.py", line 1062, in _unsloth_pre_compute_loss
return self._old_compute_loss(model, inputs, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/transformers/trainer.py", line 3708, in compute_loss
outputs = model(**inputs)
^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/accelerate/utils/operations.py", line 823, in forward
return model_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/accelerate/utils/operations.py", line 811, in __call__
return convert_to_fp32(self.model_forward(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner
return disable_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/unsloth/models/llama.py", line 1118, in PeftModelForCausalLM_fast_forward
return self.base_model(
^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 197, in forward
return self.model.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/unsloth/models/llama.py", line 970, in _CausalLM_fast_forward
causal_mask = xformers.attn_bias.LowerTriangularMask()
^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'attn_bias'
0%| | 0/60 [00:00<?, ?it/s]
and my xformers is built for py311_cu12.1.0_pyt2.5.1
Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
accelerate 1.2.1 pypi_0 pypi
aiohappyeyeballs 2.4.4 pypi_0 pypi
aiohttp 3.11.11 pypi_0 pypi
aiosignal 1.3.2 pypi_0 pypi
attrs 24.3.0 pypi_0 pypi
bitsandbytes 0.42.0 pypi_0 pypi
blas 2.16 mkl conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
ca-certificates 2024.12.14 hbcca054_0 conda-forge
certifi 2024.12.14 pypi_0 pypi
charset-normalizer 3.4.1 pypi_0 pypi
cpython 3.11.11 py311hd8ed1ab_1 conda-forge
cuda-cudart 12.1.105 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-libraries 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-opencl 12.4.127 0 nvidia
cuda-runtime 12.1.0 0 nvidia
cudatoolkit 11.7.0 hd8887f6_10 nvidia
cut-cross-entropy 24.12.3 pypi_0 pypi
datasets 3.2.0 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
docstring-parser 0.16 pypi_0 pypi
filelock 3.16.1 pyhd8ed1ab_1 conda-forge
frozenlist 1.5.0 pypi_0 pypi
fsspec 2024.9.0 pypi_0 pypi
gmp 6.3.0 hac33072_2 conda-forge
gmpy2 2.1.5 py311h0f6cedb_3 conda-forge
hf-transfer 0.1.8 pypi_0 pypi
huggingface-hub 0.27.0 pypi_0 pypi
idna 3.10 pypi_0 pypi
intel-openmp 2022.0.1 h06a4308_3633 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jinja2 3.1.5 pyhd8ed1ab_0 conda-forge
ld_impl_linux-64 2.43 h712a8e2_2 conda-forge
libblas 3.8.0 16_mkl conda-forge
libcblas 3.8.0 16_mkl conda-forge
libcublas 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufile 1.9.1.3 0 nvidia
libcurand 10.3.5.147 0 nvidia
libcusolver 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libexpat 2.6.4 h5888daf_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
libgfortran-ng 7.5.0 h14aa051_20 conda-forge
libgfortran4 7.5.0 h14aa051_20 conda-forge
libgomp 14.2.0 h77fa898_1 conda-forge
liblapack 3.8.0 16_mkl conda-forge
liblapacke 3.8.0 16_mkl conda-forge
liblzma 5.6.3 hb9d3cd8_1 conda-forge
libnpp 12.0.2.50 0 nvidia
libnsl 2.0.1 hd590300_0 conda-forge
libnvjitlink 12.1.105 0 nvidia
libnvjpeg 12.1.1.14 0 nvidia
libsqlite 3.47.2 hee588c1_0 conda-forge
libstdcxx 14.2.0 hc0a3c3a_1 conda-forge
libstdcxx-ng 14.2.0 h4852527_1 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
llvm-openmp 15.0.7 h0cdce71_0 conda-forge
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 3.0.2 py311h2dc5d0c_1 conda-forge
mdurl 0.1.2 pypi_0 pypi
mkl 2020.2 256 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mpc 1.3.1 h24ddda3_1 conda-forge
mpfr 4.2.1 h90cbb55_3 conda-forge
mpmath 1.3.0 pyhd8ed1ab_1 conda-forge
multidict 6.1.0 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
ncurses 6.5 he02047a_1 conda-forge
networkx 3.4.2 pyh267e887_2 conda-forge
numpy 2.2.1 pypi_0 pypi
openssl 3.4.0 hb9d3cd8_0 conda-forge
packaging 24.2 pypi_0 pypi
pandas 2.2.3 pypi_0 pypi
peft 0.14.0 pypi_0 pypi
pillow 11.0.0 pypi_0 pypi
pip 24.3.1 pyh8b19718_2 conda-forge
propcache 0.2.1 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
psutil 6.1.1 pypi_0 pypi
pyarrow 18.1.0 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
python 3.11.11 h9e4cc4f_1_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python_abi 3.11 5_cp311 conda-forge
pytorch 2.5.1 py3.11_cuda12.1_cudnn9.1.0_0 pytorch
pytorch-cuda 12.1 ha16c6d3_6 pytorch
pytorch-mutex 1.0 cuda pytorch
pytz 2024.2 pypi_0 pypi
pyyaml 6.0.2 py311h9ecbd09_1 conda-forge
readline 8.2 h8228510_1 conda-forge
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
rich 13.9.4 pypi_0 pypi
safetensors 0.4.5 pypi_0 pypi
scipy 1.14.1 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 75.6.0 pyhff2d567_1 conda-forge
shtab 1.7.1 pypi_0 pypi
six 1.17.0 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tokenizers 0.21.0 pypi_0 pypi
torchtriton 3.1.0 py311 pytorch
tqdm 4.67.1 pypi_0 pypi
transformers 4.47.1 pypi_0 pypi
trl 0.13.0 pypi_0 pypi
typeguard 4.4.1 pypi_0 pypi
typing_extensions 4.12.2 pyha770c72_1 conda-forge
tyro 0.9.5 pypi_0 pypi
tzdata 2024.2 pypi_0 pypi
unsloth 2024.12.12 pypi_0 pypi
unsloth-zoo 2024.12.7 pypi_0 pypi
urllib3 2.3.0 pypi_0 pypi
wheel 0.45.1 pyhd8ed1ab_1 conda-forge
xformers 0.0.28.post3 py311_cu12.1.0_pyt2.5.1 xformers
xxhash 3.5.0 pypi_0 pypi
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.18.3 pypi_0 pypi
why do I get this error?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 19 (9 by maintainers)
!pip install pip3-autoremove !pip-autoremove torch torchvision torchaudio -y !pip install torch torchvision torchaudio xformers --index-url https://download.pytorch.org/whl/cu121 !pip install unsloth !pip install evaluate
Then restart kernel
These commands worked for me!
You can try them out
just ran into the same issue. fixed with latest unsloth and xformers.
Yeah that is great. I always use no 1 and it fixes everything for me. I never needed to install xformers separately.
Apologies on the delay everyone - I most likely fixed it by now - but yes float16 and mixing bfloat16 is the issue
I also fixed the lm_head / embed_tokens issues
The notebooks should function fine now - sorry on the delay!
The Xformers issue is still a weird one - I’m unsure why that appears