unsloth: Qwen2-VL-72B 4Bit won't load

Hi!

I am having trouble loading the Qwen2-VL-72B models. Here’s a minimal code snippet, which I run on a local machine using the standard conda install from the docs. I have tried with both a 32GB Tesla-V100 and an 80GB A100.

import torch
from trl import SFTTrainer, SFTConfig
from unsloth import FastVisionModel, is_bf16_supported # FastLanguageModel for LLMs
from unsloth.trainer import UnslothVisionDataCollator

model_hf_path = 'unsloth/Qwen2-VL-72B-Instruct-bnb-4bit'
model, tokenizer = FastVisionModel.from_pretrained(
        model_hf_path, 
        load_in_4bit = True, 
        use_gradient_checkpointing = "unsloth", 
    )

I get the following error:

Traceback (most recent call last):
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
    response.raise_for_status()
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/unsloth/qwen2-vl-72b-instruct-unsloth-bnb-4bit

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 125, in _repo_and_revision_exist
    self._api.repo_info(
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 2748, in repo_info
    return method(
           ^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 2533, in model_info
    hf_raise_for_status(r)
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 454, in hf_raise_for_status
    raise _format(RepositoryNotFoundError, message, response) from e
huggingface_hub.errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-677bb76a-7e60f5736021adae14bf2f7a;410212d4-27c9-42bb-9c2d-4d27d7962042)

Repository Not Found for url: https://huggingface.co/api/models/unsloth/qwen2-vl-72b-instruct-unsloth-bnb-4bit.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/loader.py", line 413, in from_pretrained
    files = HfFileSystem(token = token).glob(os.path.join(model_name, "*.json"))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 520, in glob
    path = self.resolve_path(path, revision=kwargs.get("revision")).unresolve()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 216, in resolve_path
    _raise_file_not_found(path, err)
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/huggingface_hub/hf_file_system.py", line 1136, in _raise_file_not_found
    raise FileNotFoundError(msg) from err
FileNotFoundError: unsloth/qwen2-vl-72b-instruct-unsloth-bnb-4bit/*.json (repository not found)

Everything works perfectly with the 2B and 7B with these parameters. Furthermore, I do not get this error when I load the model with load_in_4bit = False. The model starts to load, but then I get the following error, which suggests that I am running out of RAM (even on an A100), so I would really like to load in 4bit!

Loading checkpoint shards:   0%|          | 0/31 [00:00<?, ?it/s]
Loading checkpoint shards:   3%|▎         | 1/31 [00:04<02:22,  4.75s/it]
Loading checkpoint shards:   6%|▋         | 2/31 [00:06<01:33,  3.22s/it]
Loading checkpoint shards:  10%|▉         | 3/31 [00:09<01:17,  2.77s/it]
Loading checkpoint shards:  13%|█▎        | 4/31 [00:11<01:07,  2.52s/it]
Loading checkpoint shards:  16%|█▌        | 5/31 [00:15<01:23,  3.22s/it]
Loading checkpoint shards:  19%|█▉        | 6/31 [00:20<01:37,  3.92s/it]
Loading checkpoint shards:  23%|██▎       | 7/31 [00:25<01:37,  4.06s/it]
Loading checkpoint shards:  26%|██▌       | 8/31 [00:29<01:35,  4.17s/it]
Loading checkpoint shards:  29%|██▉       | 9/31 [00:34<01:36,  4.40s/it]
Loading checkpoint shards:  32%|███▏      | 10/31 [00:38<01:30,  4.32s/it]
Loading checkpoint shards:  35%|███▌      | 11/31 [00:43<01:29,  4.47s/it]
Loading checkpoint shards:  39%|███▊      | 12/31 [00:49<01:30,  4.79s/it]
Loading checkpoint shards:  42%|████▏     | 13/31 [00:54<01:27,  4.86s/it]
Loading checkpoint shards:  45%|████▌     | 14/31 [00:59<01:25,  5.02s/it]
Loading checkpoint shards:  48%|████▊     | 15/31 [01:05<01:27,  5.45s/it]
Loading checkpoint shards:  52%|█████▏    | 16/31 [01:10<01:19,  5.29s/it]
Loading checkpoint shards:  55%|█████▍    | 17/31 [01:14<01:07,  4.84s/it]
Loading checkpoint shards:  58%|█████▊    | 18/31 [01:14<00:44,  3.43s/it]
Loading checkpoint shards:  61%|██████▏   | 19/31 [01:15<00:29,  2.45s/it]
Loading checkpoint shards:  65%|██████▍   | 20/31 [01:15<00:19,  1.76s/it]
Loading checkpoint shards:  68%|██████▊   | 21/31 [01:15<00:12,  1.28s/it]
Loading checkpoint shards:  71%|███████   | 22/31 [01:15<00:08,  1.06it/s]
Loading checkpoint shards:  74%|███████▍  | 23/31 [01:15<00:05,  1.41it/s]
Loading checkpoint shards:  77%|███████▋  | 24/31 [01:15<00:03,  1.81it/s]
Loading checkpoint shards:  81%|████████  | 25/31 [01:16<00:02,  2.28it/s]
Loading checkpoint shards:  84%|████████▍ | 26/31 [01:16<00:01,  2.78it/s]
Loading checkpoint shards:  87%|████████▋ | 27/31 [01:16<00:01,  3.27it/s]
Loading checkpoint shards:  90%|█████████ | 28/31 [01:16<00:00,  3.75it/s]
Loading checkpoint shards:  94%|█████████▎| 29/31 [01:16<00:00,  4.18it/s]
Loading checkpoint shards:  97%|█████████▋| 30/31 [01:16<00:00,  4.60it/s]
Loading checkpoint shards: 100%|██████████| 31/31 [01:17<00:00,  5.15it/s]
Loading checkpoint shards: 100%|██████████| 31/31 [01:17<00:00,  2.48s/it] 
Some parameters are on the meta device because they were offloaded to the cpu.
Traceback (most recent call last):
  File "./finetuning/src/finetune.py", line 106, in <module>
    finetune(args.llm, args.split, args.data_path, args.res_path)
  File "./finetuning/src/finetune.py", line 44, in finetune
    trainer = SFTTrainer(
              ^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/trainer.py", line 203, in new_init
    original_init(self, *args, **kwargs)
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 165, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/trl/trainer/sft_trainer.py", line 307, in __init__
    super().__init__(
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 165, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 574, in __init__
    self._move_model_to_device(model, args.device)
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/trainer.py", line 846, in _move_model_to_device
    model = model.to(device)
            ^^^^^^^^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1340, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "$HOMEminiconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 900, in _apply
    module._apply(fn)
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 900, in _apply
    module._apply(fn)
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 900, in _apply
    module._apply(fn)
  [Previous line repeated 5 more times]
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 927, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "$HOME/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1333, in convert
    raise NotImplementedError(
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Thanks in advance, and let me know if I can provide any further details.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (2 by maintainers)

Most upvoted comments

I’ll upload one for Qwen 72VL Vision!

Hi all, I found a very very hacky fix to get the large models fine-tuning, which I will document here.

Here is my full conda environment, given after running pip freeze. I am running with python=3.11. Note the earlier versions of unsloth and unsloth_zoo:

accelerate==1.4.0
aiohappyeyeballs==2.4.8
aiohttp==3.11.13
aiosignal==1.3.2
annotated-types==0.7.0
attrs==25.1.0
bitsandbytes==0.45.3
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
cut-cross-entropy==25.1.1
datasets==3.3.2
diffusers==0.32.2
dill==0.3.8
docker-pycreds==0.4.0
docstring_parser==0.16
filelock @ file:///croot/filelock_1700591183607/work
frozenlist==1.5.0
fsspec==2024.12.0
gitdb==4.0.12
GitPython==3.1.44
gmpy2 @ file:///croot/gmpy2_1738085463648/work
hf_transfer==0.1.9
huggingface-hub==0.29.1
idna==3.10
importlib_metadata==8.6.1
Jinja2 @ file:///croot/jinja2_1737760107953/work
joblib==1.4.2
markdown-it-py==3.0.0
MarkupSafe @ file:///croot/markupsafe_1738584038848/work
mdurl==0.1.2
mpmath @ file:///croot/mpmath_1690848262763/work
multidict==6.1.0
multiprocess==0.70.16
networkx @ file:///croot/networkx_1737039604450/work
numpy==2.2.3
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-cusparselt-cu12==0.6.2
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
packaging==24.2
pandas==2.2.3
peft==0.14.0
pillow==11.1.0
platformdirs==4.3.6
propcache==0.3.0
protobuf==3.20.3
psutil==7.0.0
pyarrow==19.0.1
pydantic==2.10.6
pydantic_core==2.27.2
Pygments==2.19.1
python-dateutil==2.9.0.post0
pytz==2025.1
PyYAML @ file:///croot/pyyaml_1728657952215/work
regex==2024.11.6
requests==2.32.3
rich==13.9.4
safetensors==0.5.3
scikit-learn==1.6.1
scipy==1.15.2
sentencepiece==0.2.0
sentry-sdk==2.22.0
setproctitle==1.3.5
shtab==1.7.1
six==1.17.0
smmap==5.0.2
sympy==1.13.1
threadpoolctl==3.5.0
tokenizers==0.21.0
torch==2.5.1
torchvision==0.20.1
tqdm==4.67.1
transformers==4.49.0
triton==3.1.0
trl==0.15.2
typeguard==4.4.2
typing_extensions @ file:///croot/typing_extensions_1734714854207/work
tyro==0.9.16
tzdata==2025.1
unsloth==2025.3.3
unsloth_zoo==2025.3.1
urllib3==2.3.0
wandb==0.19.8
xformers==0.0.28.post3
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0

I then made the following changes to the following files inside site-packages.

  1. I replaced unsloth_zoo/compiler.py with the file from here which prevents unsloth from hanging unnecessarily when loading.
  2. I then got the RuntimeError: only Tensors of floating point dtype can require gradients error. I went into peft_utils.py to line 202 and rewrote the def requires_grad_pre_hook(module, input) function, replacing it with:
    def requires_grad_pre_hook(module, input):
        # Modified to only set requires_grad on floating-point tensors
        if isinstance(input, torch.Tensor):
            if input.is_floating_point():
                input.requires_grad_(True)
        elif isinstance(input, (tuple, list)):
            for inp in input:
                if isinstance(inp, torch.Tensor) and inp.is_floating_point():
                    inp.requires_grad_(True)
                    break  # Set the first floating-point tensor found
        else:
            # Handle other input types if necessary
            pass
    pass

I would then occasionally get errors about not being able to import wandb or LogisticRegression, neither of which I needed, so I simply pip installed them (scikit-learn for LR). They are included in the conda environment specification above.

I can now finetune and run inference on Llama-3.2-90B and QwenVL-2.5-72B. For our specific use case, we have not identified anything out of the ordinary with respect to fine-tuning or model inference (models learn and generalize reasonably well at test time).