unsloth: [FIXED] Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

I get this error:

Traceback (most recent call last):
  File "/home/denis/Documents/ai/unsloth/llama3-chat-template.py", line 20, in <module>
    model, tokenizer = FastLanguageModel.from_pretrained(
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/unsloth/models/loader.py", line 323, in from_pretrained
    model, tokenizer = dispatch_model.from_pretrained(
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/unsloth/models/llama.py", line 1610, in from_pretrained
    tokenizer = load_correct_tokenizer(
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/unsloth/tokenizer_utils.py", line 538, in load_correct_tokenizer
    tokenizer = _load_correct_tokenizer(
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/unsloth/tokenizer_utils.py", line 496, in _load_correct_tokenizer
    fast_tokenizer = AutoTokenizer.from_pretrained(
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 897, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
    return cls._from_pretrained(
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 115, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

It works with nsloth/Llama-3.2-1B-Instruct-bnb-4bit

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 5
  • Comments: 31 (7 by maintainers)

Most upvoted comments

Oh it’s best to update transformers via pip install --upgrade "transformers>=4.45"

I am running into this same error as well when merging and exporting the 16bit model and using it on vllm. I have tried multiple models and the error is consistent. Most definitely the tokenizer exporter is broken Edit: by using latest version of docker image form vllm it now works (v0.6.2)

If I go and download the tokenizer files from the HuggingFace repository and replace them, it works.

Notebook with a working version:

@danielhanchen @katopz - here is a notebook for “offline” installation on Kaggle: (https://www.kaggle.com/code/kolyan1/offline-unsloth-package-installation-pt-2-working)

Generally one work-around is as follows:

  1. Install unsloth==2024.10.4 and torch==2.4.1
pip3 install unsloth==2024.10.4 torch==2.4.1
  1. Install transformers==4.45.2 (this will throw errors, as transformers has to be <4.45 for unsloth, but it still installs successfully and works)
pip3 install transformers==4.45.2

Updating tokenizers to latest 0.20.0 might help

I encountered the same issue as @selectorseb while deploying a finetuned Llama-3.2 model using vLLM with Docker. Initially, I faced the same problem mentioned in the original post @djannot, but after updating the vLLM Docker image, the issue was resolved.

I’m running into this same error when trying to quantize the trained models into gguf format Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3 Edit: The tokenizer unsloth exports is broken.

Thanks @danielhanchen for the fast response (as usual).

I did try this, but I now get another error:

Traceback (most recent call last):
  File "/home/denis/Documents/ai/unsloth/llama3-chat-template.py", line 113, in <module>
    trainer_stats = trainer.train()
  File "<string>", line 145, in train
  File "<string>", line 358, in _fast_inner_training_loop
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/transformers/trainer.py", line 3477, in training_step
    self.optimizer.train()
  File "/home/denis/miniconda3/envs/pytorch/lib/python3.10/site-packages/accelerate/optimizer.py", line 128, in train
    return self.optimizer.train()
AttributeError: 'AdamW' object has no attribute 'train'

Upgrading accelerate to version 0.34.0 will resolve this issue.

but when inference, it occurs that ‘ValueError: Invalid cache_implementation (dynamic). Choose one of: [‘static’, ‘offloaded_static’, ‘sliding_window’, ‘hybrid’, ‘mamba’, ‘quantized’, ‘static’]’

this error has been fixed by upgrading the unsloth to version 2024.9.post3 and transformers to version 4.45.0

but when inference, it occurs that ‘ValueError: Invalid cache_implementation (dynamic). Choose one of: [‘static’, ‘offloaded_static’, ‘sliding_window’, ‘hybrid’, ‘mamba’, ‘quantized’, ‘static’]’