LocalAI: Getting started example not working

Hi, @gitwittidbit , thanks for your feedback. I wonder if this line would work if you do not rebuild the project? Here, go-ggllm/ggllm.cpp is an bounding project, and it was loaded by make command. It should not have this kind of issue.

Yes, if you don’t rebuild the project, an error in the source/build code won’t be a problem. For me it was a problem because I needed to rebuild the project.

And if you do rebuild it and the build process stop with the error

CMake Error at tests/CMakeLists.txt:145 (set_target_properties): set_target_properties called with incorrect number of arguments.

the you can fix this by correcting line 145 in CMakeLists.txt.

Okay, I figured it out.

Mind you, OP and I stumbled across

CMake Error at tests/CMakeLists.txt:145 (set_target_properties): set_target_properties called with incorrect number of arguments.

Which happens when you rebuild local-ai (either in docker or outside) and which is set as the default behaviour.

The solution presented here so far is to avoid rebuilding local-ai. Unfortunately, that didn’t work for me as I needed to rebuild local-ai without avx2 support (which is active per default).

So, without further ado, the solution:

There is a set of double quotes missing in go-ggllm/ggllm.cpp/CMakeLists.txt in line 145:

It reads in the repo set(LLAMA_EXTRA_LIBS ${LLAMA_EXTRA_LIBS} ${ACCELERATE_FRAMEWORK})

but should read set(LLAMA_EXTRA_LIBS "${LLAMA_EXTRA_LIBS} ${ACCELERATE_FRAMEWORK}")

With this change, for me the build process goes through without the error.

Not being a software developer, unfortunately, I don’t know yet how to make a PR to correct this myself. I am going to learn that but I thought it would be better to provide the solution faster. @mudler will have this fixed in no time whereas I would probably need a couple of days to set up an IDE, git, etc. just to make the PR.

Thanks @mudler for this wonderful project! I really admire what you have created here.

-v models:/build/models
      ^             ^
      |               path inside of your container
      path in your host

The host path should be something like: /home/wouter/docker/LocalAI/models The path inside of the container should be /build/models

I get also these errors at the start of the container: unexpected end of JSON input But I ignore it currently.

@wouterverduin Can you post your complete docker command + the output ?

My current way to spin it up is:

docker run -it --rm -v ~/LLM/models:/build/models:cached -p 8080:8080 -e "ADDRESS=0.0.0.0:8080" -e "DEBUG=true" -e "CORS=true"  -e "THREADS=8" -e "REBUILD=false"  quay.io/go-skynet/local-ai:v1.20.1

You can play arrount with the -e options

Starting LocalAI using 8 threads, with models path: /build/models
unexpected end of JSON input

 ┌───────────────────────────────────────────────────┐ 
 │                   Fiber v2.47.0                   │ 
 │               http://127.0.0.1:8080               │ 
 │       (bound on host 0.0.0.0 and port 8080)       │ 
 │                                                   │ 
 │ Handlers ............ 33  Processes ........... 1 │ 
 │ Prefork ....... Disabled  PID ................. 7 │ 
 └───────────────────────────────────────────────────┘ 
[172.17.0.1]:44416  200  -  GET      /v1/models

Also to get the list of loaded models:

curl http://localhost:8080/v1/models
{"object":"list","data":[{"id":"ggml-nous-gpt4-vicuna-13b.bin","object":"model"}]}

For those trying this out on the Windows OS and facing the above issue, I got it working by following the below steps:

1. Replace the $PWD with the absolute path while mounting the volume and change the internal container path to /build/models
2. Remove the environment variable MODELS_PATH since this defaults to the mentioned value within the .env file in the repo

Sample command while using the CUDA enabled image is shared below. Note that I’ve manually downloaded the models along with the config yaml file into the ./models folder within the cloned repo.

docker run -ti --gpus all -p 8080:8080 -e THREADS=1 -v "C:\Users\XXX\YYY\LocalAI\models":/build/models localai/localai:v2.6.0-cublas-cuda12-ffmpeg-core

⚠️⚠️⚠️⚠️⚠️

Hi! I’m a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but… I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don’t engage in conversation with me, I don’t support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I am sorry but I do not know how to resolve this issue. The error message indicates that there is an issue with CMake configuration, specifically that there was an incorrect number of arguments passed to set_target_properties. Additionally, it appears that there is an issue building the ggml-transformers library, with an error occurring when trying to build the ggml.o file and another error when creating the libtransformers.a file. This could potentially be due to missing or incorrect dependencies. Without more information about the environment and any other steps taken, it is difficult to determine the cause of this issue. I suggest checking the logs for more information on the errors and possibly reviewing the installation guide and relevant documentation.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/examples/langchain-python/docker-compose.yaml
• https://github.com/go-skynet/LocalAI/tree/master/examples/rwkv/docker-compose.yaml
• https://localai.io/basics/getting_started/index.html
• https://github.com/go-skynet/LocalAI/tree/master/examples/langchain/docker-compose.yaml

Thank you guys’ feedback.

The binary inside the image is rebuilt at the start of the container to enable CPU optimizations for the execution environment, you can set the environment variable REBUILD to false to prevent this behavior.

And about the model path @finger42 already given a clear answer.