unsloth: llama.cpp GGUF breaks [FIXED]
As of 3rd December 2024 - fixed.
Please update Unsloth via
pip install --upgrade --no-deps --no-cache-dir unsloth
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 21 (3 by maintainers)
As of 3rd December 2024 - fixed.
Please update Unsloth via
pip install --upgrade --no-deps --no-cache-dir unsloth
@IridiumMaster Below are the working steps for conversion of a vanilla model to
GGUFusingUnslothandllama.cpp. The operation was performed usingRUNPODwithrunpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.041. git clone https://github.com/ggml-org/llama.cpp.git2. cd llama.cpp3. git checkout b33455. git submodule update --init --recursive6. make clean7. make all -j8. git log -1For better understanding, I’ve also attached a link to the Notebook that was used for the conversion. GGUF conversion Notebook
The issue still exists. Kindly report a fix if anyone finds it.
I got the same error. Installing Cmake in my docker container worked for me.
This is still broken for me: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [1:18:58<00:00, 11.85s/it] Saving model with quantization method: q4_k_m make: Entering directory ‘/workspace/llama.cpp’ Makefile:2: *** The Makefile build is deprecated. Use the CMake build instead. For more details, see https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md. Stop. make: Leaving directory ‘/workspace/llama.cpp’ sh: 1: cmake: not found Traceback (most recent call last): File “/workspace/unsloth-cli-v2.py”, line 235, in <module> run(args) File “/workspace/unsloth-cli-v2.py”, line 148, in run model.save_pretrained_gguf( File “/usr/local/lib/python3.11/dist-packages/unsloth/save.py”, line 1684, in unsloth_save_pretrained_gguf makefile = install_llama_cpp_make_non_blocking() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/unsloth/save.py”, line 779, in install_llama_cpp_make_non_blocking raise RuntimeError(f"*** Unsloth: Failed compiling llama.cpp using os.system(…) with error {check}. Please report this ASAP!") RuntimeError: *** Unsloth: Failed compiling llama.cpp using os.system(…) with error 32512. Please report this ASAP!
i think you can bypass by manually compiling llama.cpp (clean install) using latest cmake instructions for your system from https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#cpu-build, and then running save_pretrained_gguf.
Issues might be due to errors in cmake compilation flag for your system in https://github.com/unslothai/unsloth/blob/038e6d4c8d40207a87297ab3aaf787c19b1006d1/unsloth/save.py#L777, e.g. https://github.com/ggerganov/llama.cpp/issues/9937
I bypassed this issue but using an early version of
llama.cppand then following the below temporary solution that has now become permanent it seems.cd llama.cppgit checkout b3345git submodule update --init --recursivemake cleanmake all -jgit log -1credit goes to https://github.com/unslothai/unsloth/issues/748#issuecomment-2238395604