unsloth: AttributeError: 'NoneType' object has no attribute 'attn_bias'

I got an error of xformers:

Traceback (most recent call last):
  File "/data1/xuanyang/kooshot_cate_pred_llm_sft/sft.py", line 131, in <module>
    trainer_stats = trainer.train()
                    ^^^^^^^^^^^^^^^
  File "<string>", line 157, in train
  File "<string>", line 382, in _fast_inner_training_loop
  File "<string>", line 31, in _unsloth_training_step
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/unsloth/models/_utils.py", line 1062, in _unsloth_pre_compute_loss
    return self._old_compute_loss(model, inputs, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/transformers/trainer.py", line 3708, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/accelerate/utils/operations.py", line 823, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/accelerate/utils/operations.py", line 811, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner
    return disable_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 632, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/unsloth/models/llama.py", line 1118, in PeftModelForCausalLM_fast_forward
    return self.base_model(
           ^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 197, in forward
    return self.model.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data1/xuanyang/envs/unsloth/lib/python3.11/site-packages/unsloth/models/llama.py", line 970, in _CausalLM_fast_forward
    causal_mask = xformers.attn_bias.LowerTriangularMask()
                  ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'attn_bias'
  0%|          | 0/60 [00:00<?, ?it/s]

and my xformers is built for py311_cu12.1.0_pyt2.5.1

Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
accelerate                1.2.1                    pypi_0    pypi
aiohappyeyeballs          2.4.4                    pypi_0    pypi
aiohttp                   3.11.11                  pypi_0    pypi
aiosignal                 1.3.2                    pypi_0    pypi
attrs                     24.3.0                   pypi_0    pypi
bitsandbytes              0.42.0                   pypi_0    pypi
blas                      2.16                        mkl    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
ca-certificates           2024.12.14           hbcca054_0    conda-forge
certifi                   2024.12.14               pypi_0    pypi
charset-normalizer        3.4.1                    pypi_0    pypi
cpython                   3.11.11         py311hd8ed1ab_1    conda-forge
cuda-cudart               12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.4.127                      0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cudatoolkit               11.7.0              hd8887f6_10    nvidia
cut-cross-entropy         24.12.3                  pypi_0    pypi
datasets                  3.2.0                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
docstring-parser          0.16                     pypi_0    pypi
filelock                  3.16.1             pyhd8ed1ab_1    conda-forge
frozenlist                1.5.0                    pypi_0    pypi
fsspec                    2024.9.0                 pypi_0    pypi
gmp                       6.3.0                hac33072_2    conda-forge
gmpy2                     2.1.5           py311h0f6cedb_3    conda-forge
hf-transfer               0.1.8                    pypi_0    pypi
huggingface-hub           0.27.0                   pypi_0    pypi
idna                      3.10                     pypi_0    pypi
intel-openmp              2022.0.1          h06a4308_3633    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jinja2                    3.1.5              pyhd8ed1ab_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
libblas                   3.8.0                    16_mkl    conda-forge
libcblas                  3.8.0                    16_mkl    conda-forge
libcublas                 12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufile                 1.9.1.3                       0    nvidia
libcurand                 10.3.5.147                    0    nvidia
libcusolver               11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran-ng            7.5.0               h14aa051_20    conda-forge
libgfortran4              7.5.0               h14aa051_20    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
liblapack                 3.8.0                    16_mkl    conda-forge
liblapacke                3.8.0                    16_mkl    conda-forge
liblzma                   5.6.3                hb9d3cd8_1    conda-forge
libnpp                    12.0.2.50                     0    nvidia
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjitlink              12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libsqlite                 3.47.2               hee588c1_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
llvm-openmp               15.0.7               h0cdce71_0    conda-forge
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                3.0.2           py311h2dc5d0c_1    conda-forge
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2020.2                      256    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
mpc                       1.3.1                h24ddda3_1    conda-forge
mpfr                      4.2.1                h90cbb55_3    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_1    conda-forge
multidict                 6.1.0                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
ncurses                   6.5                  he02047a_1    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
numpy                     2.2.1                    pypi_0    pypi
openssl                   3.4.0                hb9d3cd8_0    conda-forge
packaging                 24.2                     pypi_0    pypi
pandas                    2.2.3                    pypi_0    pypi
peft                      0.14.0                   pypi_0    pypi
pillow                    11.0.0                   pypi_0    pypi
pip                       24.3.1             pyh8b19718_2    conda-forge
propcache                 0.2.1                    pypi_0    pypi
protobuf                  3.20.3                   pypi_0    pypi
psutil                    6.1.1                    pypi_0    pypi
pyarrow                   18.1.0                   pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
python                    3.11.11         h9e4cc4f_1_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python_abi                3.11                    5_cp311    conda-forge
pytorch                   2.5.1           py3.11_cuda12.1_cudnn9.1.0_0    pytorch
pytorch-cuda              12.1                 ha16c6d3_6    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2024.2                   pypi_0    pypi
pyyaml                    6.0.2           py311h9ecbd09_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
regex                     2024.11.6                pypi_0    pypi
requests                  2.32.3                   pypi_0    pypi
rich                      13.9.4                   pypi_0    pypi
safetensors               0.4.5                    pypi_0    pypi
scipy                     1.14.1                   pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                75.6.0             pyhff2d567_1    conda-forge
shtab                     1.7.1                    pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tokenizers                0.21.0                   pypi_0    pypi
torchtriton               3.1.0                     py311    pytorch
tqdm                      4.67.1                   pypi_0    pypi
transformers              4.47.1                   pypi_0    pypi
trl                       0.13.0                   pypi_0    pypi
typeguard                 4.4.1                    pypi_0    pypi
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tyro                      0.9.5                    pypi_0    pypi
tzdata                    2024.2                   pypi_0    pypi
unsloth                   2024.12.12               pypi_0    pypi
unsloth-zoo               2024.12.7                pypi_0    pypi
urllib3                   2.3.0                    pypi_0    pypi
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
xformers                  0.0.28.post3    py311_cu12.1.0_pyt2.5.1    xformers
xxhash                    3.5.0                    pypi_0    pypi
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.18.3                   pypi_0    pypi

why do I get this error?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 19 (9 by maintainers)

Most upvoted comments

!pip install pip3-autoremove !pip-autoremove torch torchvision torchaudio -y !pip install torch torchvision torchaudio xformers --index-url https://download.pytorch.org/whl/cu121 !pip install unsloth !pip install evaluate

Then restart kernel

These commands worked for me!

You can try them out

just ran into the same issue. fixed with latest unsloth and xformers.

Yeah that is great. I always use no 1 and it fixes everything for me. I never needed to install xformers separately.

Apologies on the delay everyone - I most likely fixed it by now - but yes float16 and mixing bfloat16 is the issue

I also fixed the lm_head / embed_tokens issues

The notebooks should function fine now - sorry on the delay!

The Xformers issue is still a weird one - I’m unsure why that appears