llama-cpp-python: (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')) on M2

Prerequisites

Please answer the following questions for yourself before submitting an issue.

[X ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
[X ] I carefully followed the README.md.
[ X] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
[ X] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

I’m trying to launch the Llama API following the guide in https://github.com/abetlen/llama-cpp-python#:~:text=docs/install/macos.md, on my Macbook M2.

Current Behavior

Unfortunately, after running the last command, which is

python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1

I get

RuntimeError: Failed to load shared library '/Users/alessandropaticchio/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib': dlopen(/Users/alessandropaticchio/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib, 0x0006): tried: '/Users/alessandropaticchio/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/alessandropaticchio/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib' (no such file), '/Users/alessandropaticchio/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Any help would be much appreciated!

About this issue

Original URL
State: open
Created 8 months ago
Comments: 18 (2 by maintainers)

Commits related to this issue

Add new binaries to flake.nix (#847) — committed to antoine-lizee/llama-cpp-python by longregen a year ago

Most upvoted comments

@YAY-3M-TA3 comment + using a previous version (0.2.11) worked for me.

!CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python==0.2.11

AhmadBaracat on Nov 24, 2023

Here is what worked for me after all (with latest version):

CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DLLAMA_METAL=on" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python

I started digging from the logline:

  -- CMAKE_SYSTEM_PROCESSOR: x86_64
  -- x86 detected

and stumbled upon this thread https://discourse.cmake.org/t/macos-cmake-detect-system-processor-incorrectly-on-apple-silicon/5129/5

@abetlen probably makes sense to add this in the troubleshooting / README?

muzhig on Dec 26, 2023

checking arch (0.2.17) latest version with previously working hotfix -DLLAMA_NATIVE=ON No luck 😦 I think the key is CMake config saying it detected x86:
  -- CMAKE_SYSTEM_PROCESSOR: x86_64
  -- x86 detected

Probably has something to do with llava1.5 multi-modal support being added to 0.2.14 (https://github.com/abetlen/llama-cpp-python/commit/aab74f0b2bd25cc2a9baeb743a2057dd5cada6e4) - since a second lib has to be built (libllava.dylib) - maybe this is why @steveoOn 's fix no longer works…?

YAY-3M-TA3 on Nov 17, 2023