bolt.diy: problem with ollama models

Describe the bug

ollama is corrctly configured, running, and no chat probleme were reported using the console prompt ollama.api wroking correctly with packet sender ollama is selected in Bolt ai chat web pages one of my ollama (qwen2.5-coder:0.5b (494.03M)) model is selected (!!! i have same probleme with other models) but i have the following error on my terminal

APICallError [AI_APICallError]: Not Found
    at C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@ai-sdk+provider-utils@1.0.20_zod@3.23.8\node_modules\@ai-sdk\provider-utils\src\response-handler.ts:72:16
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at postToApi (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@ai-sdk+provider-utils@1.0.20_zod@3.23.8\node_modules\@ai-sdk\provider-utils\src\post-to-api.ts:81:28)
    at OllamaChatLanguageModel.doStream (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\ollama-ai-provider@0.15.2_zod@3.23.8\node_modules\ollama-ai-provider\src\ollama-chat-language-model.ts:230:50)
    at fn (file:///C:/Users/HP/source/repos/LLMs/Bolt/node_modules/.pnpm/ai@3.4.33_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.7.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:378:23)        
    at file:///C:/Users/HP/source/repos/LLMs/Bolt/node_modules/.pnpm/ai@3.4.33_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.7.2__zod@3.23.8/node_modules/ai/core/telemetry/record-span.ts:18:22
    at _retryWithExponentialBackoff (file:///C:/Users/HP/source/repos/LLMs/Bolt/node_modules/.pnpm/ai@3.4.33_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.7.2__zod@3.23.8/node_modules/ai/util/retry-with-exponential-backoff.ts:37:12)
    at startStep (file:///C:/Users/HP/source/repos/LLMs/Bolt/node_modules/.pnpm/ai@3.4.33_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.7.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:333:13) 
    at fn (file:///C:/Users/HP/source/repos/LLMs/Bolt/node_modules/.pnpm/ai@3.4.33_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.7.2__zod@3.23.8/node_modules/ai/core/generate-text/stream-text.ts:414:11)        
    at file:///C:/Users/HP/source/repos/LLMs/Bolt/node_modules/.pnpm/ai@3.4.33_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.7.2__zod@3.23.8/node_modules/ai/core/telemetry/record-span.ts:18:22
    at chatAction (C:/Users/HP/source/repos/LLMs/Bolt/app/routes/api.chat.ts:65:20)
    at Object.callRouteAction (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+server-runtime@2.15.0_typescript@5.7.2\node_modules\@remix-run\server-runtime\dist\data.js:36:16)
    at C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:4899:19
    at callLoaderOrAction (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:4963:16)
    at async Promise.all (index 0)
    at defaultDataStrategy (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:4772:17)
    at callDataStrategyImpl (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:4835:17)
    at callDataStrategy (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:3992:19)
    at submit (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:3755:21)
    at queryImpl (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:3684:22)
    at Object.queryRoute (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+router@1.21.0\node_modules\@remix-run\router\router.ts:3629:18)
    at handleResourceRequest (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+server-runtime@2.15.0_typescript@5.7.2\node_modules\@remix-run\server-runtime\dist\server.js:402:20)
    at requestHandler (C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+server-runtime@2.15.0_typescript@5.7.2\node_modules\@remix-run\server-runtime\dist\server.js:156:18)
    at C:\Users\HP\source\repos\LLMs\Bolt\node_modules\.pnpm\@remix-run+dev@2.15.0_@remix-run+react@2.15.0_react-dom@18.3.1_react@18.3.1__react@18.3.1_typ_zyxju6yjkqxopc2lqyhhptpywm\node_modules\@remix-run\dev\dist\vite\cloudflare-proxy-plugin.js:70:25 {
  cause: undefined,
  url: 'http://127.0.0.1:11434/api/chat',
  requestBodyValues: {
    format: undefined,
    model: 'gpt-4o-mini',
    options: { num_ctx: 32768, num_predict: 8000, temperature: 0 },
    messages: [ [Object], [Object] ],
    tools: undefined
  },
  statusCode: 404,
  responseHeaders: {
    'content-length': '65',
    'content-type': 'application/json; charset=utf-8',
    date: 'Thu, 05 Dec 2024 21:01:25 GMT'
  },
  responseBody: '{"error":"model \\"gpt-4o-mini\\" not found, try pulling it first"}',
  isRetryable: false,
  data: undefined,
  [Symbol(vercel.ai.error)]: true,
  [Symbol(vercel.ai.error.AI_APICallError)]: true
}

Link to the Bolt URL that caused the error

local

Steps to reproduce

  1. download bolt from this repo
  2. extract it to your location
  3. $> pnpm install
  4. download ollama
  5. $>ollama --version
  6. $> ollama run qwen2.5-coder:0.5b
  7. go to bolt file .env.local and set OLLAMA_API_BASE_URL=http://127.0.0.1:11434

Expected behavior

no error

Screen Recording / Screenshot

Capture

Platform

  • OS:W10
  • Browser: Chrome
  • npm version : 10.5.2
  • pnpm version : 9.14.4
  • ollama version : 0.4.7

Provider Used

ollama

Model Used

qwen2.5-coder:0.5b (494.03M)

Additional context

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 22

Most upvoted comments

Ok, i figured out a little more about what is going on.

When i run the service and access it from the machine it is running on the model shows up correctly. If i access it from another machine it fails to load the qwen model into the dropdown. So maybe CORS related?

Seems like there could be a more permissive CORs setup for dev mode.

Ollama supports CORS for 127.0.0.1 and 0.0.0.0 by default. Additional origins can be configured using the OLLAMA_ORIGINS environment variable. For details, refer to How to Handle CORS Settings in OLLAMA: A Comprehensive Guide.

I´ve you dont use the latest main branch and build your docker-container again, than this is the problem, cause there was a bug which was fixed last week for the .env within docker 😉

(at least I guess this is also the problem then here)

In my case was a memory problem.

Steps to debug:

  1. service ollama stop
  2. ollama serve
  3. use bolt.diy and check ollama console

If you see this, you are getting an error 500 for overloading the memory:

**ror="model requires more system memory (11.3 GiB) than is available (9.2 GiB)"**
[GIN] 2025/01/30 - 17:33:54 **| 500 |**   31.857434ms |       127.0.0.1 | POST     "/api/chat"
time=2025-01-30T17:33:58.140+01:00 level=INFO source=server.go:104 msg="system memory" total="15.4 GiB" free="9.2 GiB" free_swap="136.0 KiB"
time=2025-01-30T17:33:58.140+01:00 level=WARN source=server.go:136 msg="model request too large for system" requested="11.3 GiB" available=9839177728 total="15.4 GiB" free="9.2 GiB" swap="136.0 KiB"
time=2025-01-30T17:33:58.140+01:00 level=INFO source=sched.go:428 msg="NewLlamaServer failed" model=/root/.ollama/models/blobs/sha256-60e05f2100071479f596b964f89f510f057ce397ea22f2833a0cfe029bfc2463 error="model requires more system memory (11.3 GiB) than is available (9.2 GiB)"
[GIN] 2025/01/30 - 17:33:58 | 500 |   30.446875ms |       127.0.0.1 | POST     "/api/chat"

Once I have used a lower model, Bolt.diy have started to receive responses from Ollama

Hope it helps

I had a version ( already ) cloned and configured 2 months ago on my windows 11 computer. Today I updated this version ( git pull origin main ), I did run “npm install” but now I have the same issue ( like everybody here ), I don’t see my list of Ollama models anymore.

I noticed that you needed to set the API Ollama url in my .env.local : OLLAMA_API_BASE_URL=http://localhost:11434

It restarted the server, but it didn’t make any difference… I also stopped my docker container, removed it and relaunched the docker command, still same issue

I’m using my local LLM models with this project, but it seems not to work anymore… 😦

I tested my Ollama url with curl and it works perfectly And still NodeJS 18.20.3 used in bolt.diy and on my computer is installed NodeJS 22.11.0

Are you running bolt in docker? You can try changing localhost to host.docker.internal, or any IP in task manager, and make sure ollama listens on 0.0.0.0

I had a version ( already ) cloned and configured 2 months ago on my windows 11 computer. Today I updated this version ( git pull origin main ), I did run “npm install” but now I have the same issue ( like everybody here ), I don’t see my list of Ollama models anymore.

I noticed that you needed to set the API Ollama url in my .env.local : OLLAMA_API_BASE_URL=http://localhost:11434

It restarted the server, but it didn’t make any difference… I also stopped my docker container, removed it and relaunched the docker command, still same issue

I’m using my local LLM models with this project, but it seems not to work anymore… 😦

I tested my Ollama url with curl and it works perfectly And still NodeJS 18.20.3 used in bolt.diy and on my computer is installed NodeJS 22.11.0

Ok, i figured out a little more about what is going on.

When i run the service and access it from the machine it is running on the model shows up correctly. If i access it from another machine it fails to load the qwen model into the dropdown. So maybe CORS related?

Seems like there could be a more permissive CORs setup for dev mode.

As long as a request from the BOLT server to the OLLAMA server fails (including the case when the OLLAMA server is not running during the BOLT server startup, as well as for all other model providers), the server will ignore the model settings in the browser and directly use the default model claude-3-5-sonnet-latest for the conversation.

My PATCH fixed this issue, ensuring that the server always follows the model settings from the browser when making requests. However, if the browser also cannot access the OLLAMA server, the OLLAMA model cannot be selected. This is an exceptional case, and the server should ideally throw an exception. But for compatibility, I chose to retain the error behavior where the server defaults to selecting claude-3-5-sonnet-latest.

Since the main branch is focused on making changes to unimportant UI elements, I didn’t have time to follow up with further commits, so the PR has been closed.

I can confirm that today this is still happening even with the latest commits.