unsloth: RuntimeError: Unsloth: The file 'llama.cpp/llama-quantize' or 'llama.cpp/quantize' does not exist

The below error occured while trying to convert model to gguf format.

I noticed that quantized folder resides in llama.cpp/examples/quantize

RuntimeError: Unsloth: The file 'llama.cpp/llama-quantize' or 'llama.cpp/quantize' does not exist.
But we expect this file to exist! Maybe the llama.cpp developers changed the name?

# Save to q4_k_m GGUF
if True: model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 17
  • Comments: 40 (10 by maintainers)

Most upvoted comments

It should function - are you using Colab?

Well, mine is as follows: NVIDIA V100 Driver Version: 535.146.02 CUDA Version: 12.1

I temporarily solved this problem by rolling back llama.cpp

cd llama.cpp git checkout b3345 git submodule update --init --recursive make clean make all -j git log -1

I got this issue on ubuntu, and the following steps worked for me.

  1. Build manually with cmake (It seems that make does not work anymore.), following the llama.cpp build instruction.
  2. The above creates executable files (llama-*) under llama.cpp/build/bin. Copy them over to directly under llama.cpp/.
  3. Re-run the failed unsloth call. (In my case, push_to_hub_gguf.)

@Zhangy-ly That is an effective workaround.

cd llama.cpp git checkout b3345 git submodule update --init --recursive make clean make all -j git log -1

SUPER DUMB EASY WORKAROUND FOR WINDOWS USERS:

just move the binaries from the latest llama build .zip into the base llama.cpp folder. don’t need to build anything yourself

get em from here: https://github.com/ggml-org/llama.cpp/releases

i used llama-b4951-bin-win-cuda-cu12.4-x64.zip (since i’m using cuda toolkit 12.4) and it works just fine


also in “Lib\site-packages\unsloth\save.py” i had to change:

command = f"./{quantize_location} {full_precision_location} "\
                f"{final_location} {quant_method} {n_cpus}"

to: command = f"{Path(quantize_location).absolute()} {full_precision_location} {final_location} {quant_method} {n_cpus}"

or else it’ll tell me '.' is not recognized as an internal or external command, operable program or batch file.

edit: btw i made a full on script to just set everything up for unsloth from scratch if you’re ever feeling stuck (comes with an example notebook and dataset too): https://github.com/hoffor/llms-are-dumb

I got this issue on ubuntu, and the following steps worked for me.

  1. Build manually with cmake (It seems that make does not work anymore.), following the llama.cpp build instruction.
  2. The above creates executable files (llama-*) under llama.cpp/build/bin. Copy them over to directly under llama.cpp/.
  3. Re-run the failed unsloth call. (In my case, push_to_hub_gguf.)

It works for me.

In Collab env i used

!(cd llama.cpp; cmake -B build;cmake --build build --config Release)

Then i copied the executable to /content/llama.cpp directory with cp then i re-ran the celd.

manually make works. it generates llama.cpp/llama-quantize

$ cd llama.cpp
make

It looks we need to first run make on llama cpp folder manually, not sure why it stopped working in unsloth ggerganov/llama.cpp#8107 if you run make command in llama.cpp folder it will work

Same problem here. This tip solved the issue.

$ cd llama.cpp
make

A bit of a noob here, but I have a workaround. I had built llama.cpp with VS2022 using cmake. I had a llama.cpp\bin\Releases with the resulting dll and exe files, which unsloth couldn’t find. Simply copying that whole folder to llama.cpp\llama-quantize worked. I was initially confused as to what exactly unsloth was looking for.

Summary: Just copy llama.cpp/build/bin/llama-quantize to llama.cpp/ , then try again.

I got this issue on ubuntu, and the following steps worked for me.

  1. Build manually with cmake (It seems that make does not work anymore.), following the llama.cpp build instruction.
  2. The above creates executable files (llama-*) under llama.cpp/build/bin. Copy them over to directly under llama.cpp/.
  3. Re-run the failed unsloth call. (In my case, push_to_hub_gguf.)

UPD 10.02.2025:

This is my recept for fix this bug ONLY FOR WINDOWS 11. (I’m using Windows 11 with WSL Ubuntu)

if you wanna full solution check my fix: https://github.com/unslothai/unsloth/pull/1649

  1. clone official repo of llama.cpp.
  2. build with cmake and be sure, that after this you have llama-quantize.exe in your folder https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#cpu-build
  3. for successful saving .gguf model, you need to edit file site-packages/unsloth/save.py where you have codelines like this:

llama-quantize to llama-quantize.exe

quantize_location = None if os.path.exists("llama.cpp/quantize.exe"): quantize_location = "llama.cpp/quantize.exe" elif os.path.exists("llama.cpp/llama-quantize.exe"): quantize_location = "llama.cpp/llama-quantize.exe"

and this working for me.

Image

@danielhanchen yes! It works. 👍

It looks we need to first run make on llama cpp folder manually, not sure why it stopped working in unsloth ggerganov/llama.cpp#8107 if you run make command in llama.cpp folder it will work

Same problem here. This tip solved the issue.

$ cd llama.cpp
make

@Zhangy-ly That is an effective workaround.

cd llama.cpp git checkout b3345 git submodule update --init --recursive make clean make all -j git log -1

To anyone having error while using those bash commands, use: ! before each command

It looks we need to first run make on llama cpp folder manually, not sure why it stopped working in unsloth https://github.com/ggerganov/llama.cpp/issues/8107 if you run make command in llama.cpp folder it will work

Hello, on my ubuntu server I have a new error due to export the model with llama.cpp. I copied llama-quantize to the main folder llama.cpp but now I get a new error when I try to export the model with: model.save_pretrained_gguf(“mymodel”, tokenizer, quantization_method = “q4_k_m”, maximum_memory_usage=0.7)

line 824, in try_execute raise RuntimeError(f"*** Unsloth: Failed compiling llama.cpp with {line}. Please report this ASAP!") RuntimeError: *** Unsloth: Failed compiling llama.cpp with WARNING:hf-to-gguf:************************************************************************************** . Please report this ASAP!

now im very confused because it works one and now I have really no idea how to fix this. But this is just for the phi4 model

Sorry on the issues on llama.cpp 😦 I might actually make a section with exact details on how to do llama.cpp properly

@Zhangy-ly That is an effective workaround.

cd llama.cpp git checkout b3345 git submodule update --init --recursive make clean make all -j git log -1

solved my situation. There are no llama-quantize and quantize file in the newest git source(08/07/2024). So, unslothai should install the specific version of llama.cpp to fix this issue. Thank you! 😉