unsloth: llama.cpp GGUF breaks [FIXED]

As of 3rd December 2024 - fixed.

Please update Unsloth via

pip install --upgrade --no-deps --no-cache-dir unsloth

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 21 (3 by maintainers)

Most upvoted comments

@IridiumMaster Below are the working steps for conversion of a vanilla model to GGUF using Unsloth and llama.cpp. The operation was performed using RUNPOD with runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04

1. git clone https://github.com/ggml-org/llama.cpp.git

2. cd llama.cpp

3. git checkout b3345

5. git submodule update --init --recursive

6. make clean

7. make all -j

8. git log -1


For better understanding, I’ve also attached a link to the Notebook that was used for the conversion. GGUF conversion Notebook

The issue still exists. Kindly report a fix if anyone finds it.

This is still broken for me: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [1:18:58<00:00, 11.85s/it] Saving model with quantization method: q4_k_m make: Entering directory ‘/workspace/llama.cpp’ Makefile:2: *** The Makefile build is deprecated. Use the CMake build instead. For more details, see https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md. Stop. make: Leaving directory ‘/workspace/llama.cpp’ sh: 1: cmake: not found Traceback (most recent call last): File “/workspace/unsloth-cli-v2.py”, line 235, in run(args) File “/workspace/unsloth-cli-v2.py”, line 148, in run model.save_pretrained_gguf( File “/usr/local/lib/python3.11/dist-packages/unsloth/save.py”, line 1684, in unsloth_save_pretrained_gguf makefile = install_llama_cpp_make_non_blocking() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/unsloth/save.py”, line 779, in install_llama_cpp_make_non_blocking raise RuntimeError(f"*** Unsloth: Failed compiling llama.cpp using os.system(…) with error {check}. Please report this ASAP!") RuntimeError: *** Unsloth: Failed compiling llama.cpp using os.system(…) with error 32512. Please report this ASAP!

I got the same error. Installing Cmake in my docker container worked for me.

This is still broken for me: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [1:18:58<00:00, 11.85s/it] Saving model with quantization method: q4_k_m make: Entering directory ‘/workspace/llama.cpp’ Makefile:2: *** The Makefile build is deprecated. Use the CMake build instead. For more details, see https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md. Stop. make: Leaving directory ‘/workspace/llama.cpp’ sh: 1: cmake: not found Traceback (most recent call last): File “/workspace/unsloth-cli-v2.py”, line 235, in <module> run(args) File “/workspace/unsloth-cli-v2.py”, line 148, in run model.save_pretrained_gguf( File “/usr/local/lib/python3.11/dist-packages/unsloth/save.py”, line 1684, in unsloth_save_pretrained_gguf makefile = install_llama_cpp_make_non_blocking() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/unsloth/save.py”, line 779, in install_llama_cpp_make_non_blocking raise RuntimeError(f"*** Unsloth: Failed compiling llama.cpp using os.system(…) with error {check}. Please report this ASAP!") RuntimeError: *** Unsloth: Failed compiling llama.cpp using os.system(…) with error 32512. Please report this ASAP!

i think you can bypass by manually compiling llama.cpp (clean install) using latest cmake instructions for your system from https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#cpu-build, and then running save_pretrained_gguf.

Issues might be due to errors in cmake compilation flag for your system in https://github.com/unslothai/unsloth/blob/038e6d4c8d40207a87297ab3aaf787c19b1006d1/unsloth/save.py#L777, e.g. https://github.com/ggerganov/llama.cpp/issues/9937

I bypassed this issue but using an early version of llama.cpp and then following the below temporary solution that has now become permanent it seems. cd llama.cpp

git checkout b3345

git submodule update --init --recursive

make clean

make all -j

git log -1


credit goes to https://github.com/unslothai/unsloth/issues/748#issuecomment-2238395604