unsloth: Unable to load unsloth trained model saved to a local directory.
I created a tar file out of a unsloth fine-tuned model(base-model: unsloth/gemma-2b-bnb-4bit) using PEFT and pushed it to gcsBucket. I am downloading the artifacts from gcs bucket, extracting the files and locally saving them into a folder. Now, I am trying to load the model using this local directory path, but getting error : Exception: expected value at line 1 column 1
model, tokenizer = FastLanguageModel.from_pretrained(
model_name ='./model',
max_seq_length = max_seq_length,
dtype = None,
load_in_4bit = True,
local_files_only = True,
)
FastLanguageModel.for_inference(model)
How can I load the model from the given directory?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (3 by maintainers)
Hello ,
You should now be seeing improvements with the latest version of unsloth. Please make sure to upgrade your installation to the latest version using
will close this for now. Feel free to comment back with details and code that would help us reproduce the issue if you’re still experiencing the same issue even after updating to the latest version. Thank you for using unsloth
This worked for me, but first, install this dependency :
!pip install bitsandbytesSorry been a bit busy apologies! Weird it should normally work since I detect a local inventory - how about passing just
"model"and not"./model"Ok will try investigating this - sorry on the issue!
Yes, HF loading works fine. The key observation is : even loading from local works but its not working when a tar file is created from the model artifacts and is extracted back before loading.