llama-cpp-python: Error: Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES=1

Hey, I’m not sure if this is a bug, or I just need change a config somewhere, but since the update, I get this error when I try to run models split across two cuda devices:

ValueError: Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES=1

Is that related to this? https://github.com/ggerganov/llama.cpp/pull/5240

Here is the command I ran: python3 -m llama_cpp.server --model /Models/deepseek-coder-33b-instruct.Q4_K_M.gguf --n_gpu_layers 56 --tensor_split 64 36 --offload_kqv false --n_ctx 8000 --n_batch 56 --chat_format chatml

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (4 by maintainers)

Most upvoted comments

I have just got this error with version 0.2.78. Previous versions (.77, .76, .75) worked fine.

At the office we’re having the same issue.

At first, manually building didn’t work so we moved to the WHL prebuilt. Which worker for version 0.2.81 But now, with latest prebuilt package, it doesn’t work anymore.

pip install llama-cpp-python==0.2.81 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124 --upgrade --force-reinstall

Later we run the following commands :

python -c "import llama_cpp; print(llama_cpp.llama_max_devices())"
16
python -c "import llama_cpp; print(llama_cpp.llama_supports_gpu_offload())"
True

And we can see GPU support works.

Later on with latest prebuilt wheel (and even the manual built one) :

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122 --upgrade

python -c "import llama_cpp; print(llama_cpp.llama_max_devices())"
1
python -c "import llama_cpp; print(llama_cpp.llama_supports_gpu_offload())"
False

Now we can see GPU support is “dead”. Only from this version and above.

Interesting, I built my docker container from code, the key step is RUN CUDACXX=/usr/local/cuda-12/bin/nvcc CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip3 install -e .[server] with the built-out container, I encountered the LLAMA_MAX_DEVICES = 1 issue.

I modified this line this morning, change it to RUN CUDACXX=/usr/local/cuda-12/bin/nvcc CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip3 install -e ., it works!