unsloth: RuntimeError: Unsloth: The file 'llama.cpp/llama-quantize' or 'llama.cpp/quantize' does not exist
The below error occured while trying to convert model to gguf format.
I noticed that quantized folder resides in llama.cpp/examples/quantize
RuntimeError: Unsloth: The file 'llama.cpp/llama-quantize' or 'llama.cpp/quantize' does not exist.
But we expect this file to exist! Maybe the llama.cpp developers changed the name?
# Save to q4_k_m GGUF
if True: model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 17
- Comments: 40 (10 by maintainers)
Well, mine is as follows: NVIDIA V100 Driver Version: 535.146.02 CUDA Version: 12.1
I temporarily solved this problem by rolling back llama.cpp
cd llama.cppgit checkout b3345git submodule update --init --recursivemake cleanmake all -jgit log -1I got this issue on ubuntu, and the following steps worked for me.
@Zhangy-ly That is an effective workaround.
SUPER DUMB EASY WORKAROUND FOR WINDOWS USERS:
just move the binaries from the latest llama build .zip into the base llama.cpp folder. don’t need to build anything yourself
get em from here: https://github.com/ggml-org/llama.cpp/releases
i used llama-b4951-bin-win-cuda-cu12.4-x64.zip (since i’m using cuda toolkit 12.4) and it works just fine
also in “Lib\site-packages\unsloth\save.py” i had to change:
to:
command = f"{Path(quantize_location).absolute()} {full_precision_location} {final_location} {quant_method} {n_cpus}"or else it’ll tell me
'.' is not recognized as an internal or external command, operable program or batch file.edit: btw i made a full on script to just set everything up for unsloth from scratch if you’re ever feeling stuck (comes with an example notebook and dataset too): https://github.com/hoffor/llms-are-dumb
It works for me.
In Collab env i used
Then i copied the executable to /content/llama.cpp directory with cp then i re-ran the celd.
manually
makeworks. it generatesllama.cpp/llama-quantizeA bit of a noob here, but I have a workaround. I had built llama.cpp with VS2022 using cmake. I had a llama.cpp\bin\Releases with the resulting dll and exe files, which unsloth couldn’t find. Simply copying that whole folder to llama.cpp\llama-quantize worked. I was initially confused as to what exactly unsloth was looking for.
Summary: Just copy
llama.cpp/build/bin/llama-quantizetollama.cpp/, then try again.UPD 10.02.2025:
This is my recept for fix this bug ONLY FOR WINDOWS 11. (I’m using Windows 11 with WSL Ubuntu)
if you wanna full solution check my fix: https://github.com/unslothai/unsloth/pull/1649
llama-quantize to llama-quantize.exequantize_location = None if os.path.exists("llama.cpp/quantize.exe"): quantize_location = "llama.cpp/quantize.exe" elif os.path.exists("llama.cpp/llama-quantize.exe"): quantize_location = "llama.cpp/llama-quantize.exe"and this working for me.
@danielhanchen yes! It works. 👍
@Antonytm Would https://github.com/unslothai/unsloth/wiki#manually-saving-to-gguf be helpful? Sorry on the delay!
Same problem here. This tip solved the issue.
To anyone having error while using those
bashcommands, use:!before each commandIt looks we need to first run make on llama cpp folder manually, not sure why it stopped working in unsloth https://github.com/ggerganov/llama.cpp/issues/8107 if you run make command in llama.cpp folder it will work
Hello, on my ubuntu server I have a new error due to export the model with llama.cpp. I copied llama-quantize to the main folder llama.cpp but now I get a new error when I try to export the model with: model.save_pretrained_gguf(“mymodel”, tokenizer, quantization_method = “q4_k_m”, maximum_memory_usage=0.7)
line 824, in try_execute raise RuntimeError(f"*** Unsloth: Failed compiling llama.cpp with {line}. Please report this ASAP!") RuntimeError: *** Unsloth: Failed compiling llama.cpp with WARNING:hf-to-gguf:************************************************************************************** . Please report this ASAP!
now im very confused because it works one and now I have really no idea how to fix this. But this is just for the phi4 model
Sorry on the issues on llama.cpp 😦 I might actually make a section with exact details on how to do llama.cpp properly
solved my situation. There are no llama-quantize and quantize file in the newest git source(08/07/2024). So, unslothai should install the specific version of llama.cpp to fix this issue. Thank you! 😉