bolt.diy: Unable to use any Ollama models, although Ollama itself is working fine.
Describe the bug
I have Ollama installed in Windows 11 24H2, default port 11434.
ollama list
NAME ID SIZE MODIFIED
opencoder-extra:8b a8a4a23defc6 4.7 GB 5 seconds ago
opencoder:8b c320df6c224d 4.7 GB 10 minutes ago
granite3-dense-extra:8b 2763b99345c8 4.9 GB 8 hours ago
granite3-dense:8b b5e91128f3ef 4.9 GB 21 hours ago
llama3.2:latest a80c4f17acd5 2.0 GB 2 weeks ago
llava-phi3:latest c7edd7b87593 2.9 GB 2 weeks ago
mxbai-embed-large:latest 468836162de7 669 MB 2 weeks ago
mistral-nemo:latest 994f3b8b7801 7.1 GB 2 weeks ago
qwen2.5:7b 845dbda0ea48 4.7 GB 2 weeks ago
ollama serve
2024/11/12 18:30:19 routes.go:1189: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\Marhensa\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-11-12T18:30:19.019+07:00 level=INFO source=images.go:755 msg="total blobs: 38"
time=2024-11-12T18:30:19.025+07:00 level=INFO source=images.go:762 msg="total unused blobs removed: 0"
time=2024-11-12T18:30:19.031+07:00 level=INFO source=routes.go:1240 msg="Listening on 127.0.0.1:11434 (version 0.4.1)"
time=2024-11-12T18:30:19.033+07:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 cuda_v11 cuda_v12 rocm cpu]"
time=2024-11-12T18:30:19.033+07:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-11-12T18:30:19.033+07:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2024-11-12T18:30:19.033+07:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=6 efficiency=0 threads=12
time=2024-11-12T18:30:19.301+07:00 level=INFO source=types.go:123 msg="inference compute" id=GPU-025d3e54-b313-bf37-470b-5f87afd418ee library=cuda variant=v12 compute=8.6 driver=12.7 name="NVIDIA GeForce RTX 3060"
I install bolt.new-any-llm on WSL2 Debian 12 (mirrored network mode)
I can access and use Ollama just fine from WSL2 Debian bash terminal
wsluser@DESKTOP:~/labs/bolt.new-any-llm$ curl http://127.0.0.1:11434
Ollama is running
I can even try to use it to ask “why is the sky blue?” question from WSL2 Debian bash using curl, so the network / connection is not a problem.
wsluser@DESKTOP:~/labs/bolt.new-any-llm$ curl -D - http://127.0.0.1:11434/api/chat -d '{"model":"opencoder-extra:8b","messages":[{"role":"user","content":"why is the sky blue?"}],"stream":false}'
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 12 Nov 2024 11:47:08 GMT
Content-Length: 740
{"model":"opencoder-extra:8b","created_at":"2024-11-12T11:47:08.0489403Z","message":{"role":"assistant","content":"This isn't actually a problem to solve through programming per se. The color of the sky can vary depending on the time of year and location. In general terms, it's due to scattering from millions of tiny particles in the atmosphere, referred to as Rayleigh scattering. This effect makes the sky appear blue when viewed from above. But this could also be influenced by factors like atmospheric conditions or even human eye adaptation.\n"},"done_reason":"stop","done":true,"total_duration":7165754700,"load_duration":2206961600,"prompt_eval_count":31,"prompt_eval_duration":570000000,"eval_count":86,"eval_duration":4295000000}
I set the .env.local for Ollama base URL with http://127.0.0.1:11434
I run the container with that .env.local file
the command is docker compose --profile development --env-file .env.local up
wsluser@DESKTOP:~/labs/bolt.new-any-llm$ docker compose --profile development --env-file .env.local up
[+] Running 1/0
✔ Container boltnew-any-llm-bolt-ai-dev-1 Created 0.0s
Attaching to bolt-ai-dev-1
bolt-ai-dev-1 |
bolt-ai-dev-1 | > bolt@ dev /app
bolt-ai-dev-1 | > remix vite:dev "--host" "0.0.0.0"
bolt-ai-dev-1 |
bolt-ai-dev-1 | ➜ Local: http://localhost:5173/
bolt-ai-dev-1 | ➜ Network: http://172.18.0.2:5173/
the container runs fine, I opened up browser the Bolt can see the Ollama LLM lists:
but when the conversation starts, it gives error RIGHT AWAY.
in bash console there error accessing Ollama:
bolt-ai-dev-1 | > bolt@ dev /app
bolt-ai-dev-1 | > remix vite:dev "--host" "0.0.0.0"
bolt-ai-dev-1 |
bolt-ai-dev-1 | ➜ Local: http://localhost:5173/
bolt-ai-dev-1 | ➜ Network: http://172.18.0.2:5173/
bolt-ai-dev-1 | RetryError [AI_RetryError]: Failed after 3 attempts. Last error: Cannot connect to API: connect ECONNREFUSED 127.0.0.1:11434
bolt-ai-dev-1 | at _retryWithExponentialBackoff (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:98:13)
bolt-ai-dev-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
bolt-ai-dev-1 | at async startStep (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:3903:13)
bolt-ai-dev-1 | at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:3977:11)
bolt-ai-dev-1 | at async file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:256:22
bolt-ai-dev-1 | at async chatAction (/app/app/routes/api.chat.ts:49:20)
bolt-ai-dev-1 | at async Object.callRouteAction (/app/node_modules/.pnpm/@remix-run+server-runtime@2.10.0_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/data.js:37:16)
bolt-ai-dev-1 | at async /app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4612:21
bolt-ai-dev-1 | at async callLoaderOrAction (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4677:16)
bolt-ai-dev-1 | at async Promise.all (index 1)
bolt-ai-dev-1 | at async callDataStrategyImpl (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4552:17)
bolt-ai-dev-1 | at async callDataStrategy (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4041:19)
bolt-ai-dev-1 | at async submit (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:3900:21)
bolt-ai-dev-1 | at async queryImpl (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:3858:22)
bolt-ai-dev-1 | at async Object.queryRoute (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:3827:18)
bolt-ai-dev-1 | at async handleResourceRequest (/app/node_modules/.pnpm/@remix-run+server-runtime@2.10.0_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/server.js:413:20)
bolt-ai-dev-1 | at async requestHandler (/app/node_modules/.pnpm/@remix-run+server-runtime@2.10.0_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/server.js:156:18)
bolt-ai-dev-1 | at async /app/node_modules/.pnpm/@remix-run+dev@2.10.0_@remix-run+react@2.10.2_react-dom@18.3.1_react@18.3.1__react@18.3.1_typ_qwyxqdhnwp3srgtibfrlais3ge/node_modules/@remix-run/dev/dist/vite/cloudflare-proxy-plugin.js:70:25 {
bolt-ai-dev-1 | cause: undefined,
bolt-ai-dev-1 | reason: 'maxRetriesExceeded',
bolt-ai-dev-1 | errors: [
bolt-ai-dev-1 | APICallError [AI_APICallError]: Cannot connect to API: connect ECONNREFUSED 127.0.0.1:11434
bolt-ai-dev-1 | at postToApi (/app/node_modules/.pnpm/@ai-sdk+provider-utils@1.0.20_zod@3.23.8/node_modules/@ai-sdk/provider-utils/dist/index.js:446:15)
bolt-ai-dev-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
bolt-ai-dev-1 | at async OllamaChatLanguageModel.doStream (/app/node_modules/.pnpm/ollama-ai-provider@0.15.2_zod@3.23.8/node_modules/ollama-ai-provider/dist/index.js:485:50)
bolt-ai-dev-1 | at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:3938:23)
bolt-ai-dev-1 | at async file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:256:22
bolt-ai-dev-1 | at async _retryWithExponentialBackoff (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:86:12)
bolt-ai-dev-1 | at async startStep (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:3903:13)
bolt-ai-dev-1 | at async fn (file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:3977:11)
bolt-ai-dev-1 | at async file:///app/node_modules/.pnpm/ai@3.4.9_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.30_typescript@5.5.2__zod@3.23.8/node_modules/ai/dist/index.mjs:256:22
bolt-ai-dev-1 | at async chatAction (/app/app/routes/api.chat.ts:49:20)
bolt-ai-dev-1 | at async Object.callRouteAction (/app/node_modules/.pnpm/@remix-run+server-runtime@2.10.0_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/data.js:37:16)
bolt-ai-dev-1 | at async /app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4612:21
bolt-ai-dev-1 | at async callLoaderOrAction (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4677:16)
bolt-ai-dev-1 | at async Promise.all (index 1)
bolt-ai-dev-1 | at async callDataStrategyImpl (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4552:17)
bolt-ai-dev-1 | at async callDataStrategy (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:4041:19)
bolt-ai-dev-1 | at async submit (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:3900:21)
bolt-ai-dev-1 | at async queryImpl (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:3858:22)
bolt-ai-dev-1 | at async Object.queryRoute (/app/node_modules/.pnpm/@remix-run+router@1.17.0/node_modules/@remix-run/router/dist/router.cjs.js:3827:18)
bolt-ai-dev-1 | at async handleResourceRequest (/app/node_modules/.pnpm/@remix-run+server-runtime@2.10.0_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/server.js:413:20)
bolt-ai-dev-1 | at async requestHandler (/app/node_modules/.pnpm/@remix-run+server-runtime@2.10.0_typescript@5.5.2/node_modules/@remix-run/server-runtime/dist/server.js:156:18)
bolt-ai-dev-1 | at async /app/node_modules/.pnpm/@remix-run+dev@2.10.0_@remix-run+react@2.10.2_react-dom@18.3.1_react@18.3.1__react@18.3.1_typ_qwyxqdhnwp3srgtibfrlais3ge/node_modules/@remix-run/dev/dist/vite/cloudflare-proxy-plugin.js:70:25 {
bolt-ai-dev-1 | cause: [Error],
bolt-ai-dev-1 | url: 'http://127.0.0.1:11434/api/chat',
bolt-ai-dev-1 | requestBodyValues: [Object],
bolt-ai-dev-1 | statusCode: undefined,
bolt-ai-dev-1 | responseHeaders: undefined,
bolt-ai-dev-1 | responseBody: undefined,
bolt-ai-dev-1 | isRetryable: true,
bolt-ai-dev-1 | data: undefined,
bolt-ai-dev-1 | [Symbol(vercel.ai.error)]: true,
bolt-ai-dev-1 | [Symbol(vercel.ai.error.AI_APICallError)]: true
bolt-ai-dev-1 | },
it cannot access http://127.0.0.1:11434/api/chat ? or what?
is it POST vs GET thingy that Ollama have problem with?
Link to the Bolt URL that caused the error
Steps to reproduce
- Install Ollama on Windows, get some models, serve it to default port
- Set WSL2 Debian 12 in mirrored networking mode, so Debian could access resources from Windows (like
127.0.0.1:11434) inside Linux WSL2. - Set the Ollama in
.env.local - Set Docker container, using development profile with
docker compose --profile development --env-file .env.local up - Open the browser, choose Ollama, choose the model
- Ask something
- ERROR
Expected behavior
Error.
Screen Recording / Screenshot
https://github.com/user-attachments/assets/a3f6d9f8-6f43-4c56-a9c8-40d550f3e6bb
Platform
- OS: Windows (for Ollama), WSL2 Debian 12 Mirrored Network Mode (for this to run from Docker)
- Browser: Edge
- Version:
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 8
- Comments: 51
same here, pulled the latest version and i have my OLLAMA_BASE_URL set properly, problem is when choosing Ollama, i can see my downloaded models, but for some reason when you hit enter to code something, the app si trying to get “claude-3-5-sonnet-latest” instead of the selected Ollama model:
later edit with additional information:
In dev tools in browser (latest Chrome) the payload sent to the /api/chat:
So the model and provider are properly sent to the app. This is picked up by
Chat.client.tsxline 206 i think.Also, from the error it seems that the provider is properly selected but not the model, and it defaults to DEFAULT_MODEL which is claude-3-5-sonnet-latest.
LATER EDIT 2:
Problem is in PR #188 which keeps the user choice and introduces regex match for provider and model. I added some console.log entries in stream-text.ts file and here is the output while choosing Ollama and code-qwen2.5 in the frontend:
more debug:
REGEX is failing because the MODEL_LIST doesn’t contain the OLLAMA models (i have 2 of them: qwen2.5-coder:7b and llama-3.1:latest) which are not in the list, hence the REGEX match failing and app defaulting to DEFAULT_MODEL and DEFAULT_PROVIDER.
Well… me I have this issue but I manage to hard fix the issue of running local Ollama at localhost/127.0.0.1:11434 … there are some bugs in current project preventing local use
Not sure why I’m getting always the claude-sonnet model instead of the model selected in the dropdown, for me a custom llama3.1:8b with the ctx windows of 32k by now
First set your model at app/lib/.server/llm/stream-text.ts in the method streamText, my case I set :
currentModel = 'llama3.1:8b';just before the returnSecond, not sure why but it’s attempting to connect to IPV6 localhost instead of IPV4, so I’m hardcoding in the getOllamaModel method @ app/lib/.server/llm/model.ts :
Ollama.config.baseURL = 'http://127.0.0.1:11434/api';And that’s it, that’s my hardfix which allow me to make it work by now, created the fork and PR just in case you might want to pull from my fork
workaround: if you are running in docker and you use an external URL for OLLAMA, just set in docker-compose.yaml ‘RUNNING_IN_DOCKER=false’ otherwise the app will silently use host.docker.internal URL in this function here:
So if RUNNING_IN_DOCKER is true, the Base URL for Ollama in the BACKEND will be set to
host.docker.internaland the model list pull will silently fail with no warning or errors. In the FRONTEND however, it will be set to whatever you added in the .env.local file as OLLAMA_BASE_API_URL.I don’t know or found out where is that happening
This is a major inconsistency of the app. The BASE URL should be retrieved from
.env.localto keep the application consistent, and only add instructions to the README file for users that needhost.docker.internalas an additional dns entry in docker-compose file.finally 🥹
https://github.com/user-attachments/assets/413eba94-06ec-459e-bac3-427209505926
Ok so using @dctfor 's fixed branch and adding to the docker compose:
network_mode: "host"I am finally able to get responses from chatting. I did not try network mode host before trying his branch.
And now is fixed on my PR, with that change virtually will allow any model from the source you set in the env vars, no issue with this simple change on my local.
Thanks for the info, Ollama is up top for me in terms of provider issue resolution; going to spend some time with these issues starting tomorrow
How are people getting their models for ollama to show up in the drop down? Hahaha. I can’t even get that far. I was able to deploy OpenHands and use my Ollama instance with no issues at all.
As i see in your screenshot, the
.env.localis not passed to the container since it is added it in.dockerignorefile by default. You need to remove it from there so the file will be copied to the actual docker container, andbindings.shneeds that file.Also, make sure you are running this in the same docker network as ollama and webui containers. If it is so, you need to get the container IP address for Ollama container ,
docker inspect ollama_container_nameit will be something like172.19.x.xand you actually need to pass that URL as OLLAMA_BASE_URL in your bolt setup in.env.local.All Ollama models can be listed in oTToDev, but they fail whenever a conversation starts, all of them.
I am having a similar issue where the app sees the Ollama models, but the chat fails. What model is it trying to send the request to? For me, no matter what Ollama model I chose I always tries to send the request to Claude 3.5-Sonnet which makes it fail. Same problem when I try to use LMStudio. All local LLMs appear to be broken.
I am also using docker, and I have the same issue as OP. Unfortunately, I was not able to get it running using the change to OLLAMA_API_BASE_URL. In fact, that doesn’t make sense to me because, as OP mentioned, this command works
curl -D - http://127.0.0.1:11434/api/chat -d '{"model":"qwen2.5-coder-extra-ctx:7b","messages":[{"role":"user","content":"why is the sky blue?"}],"stream":false}@dctfor That hardcoded fix works for me as well. I appreciate you figuring this out and sharing it.
At least I can get something done until this is being addressed. Thanks a lot!
I’m one of the people who has seen it work, but most of the time my list is empty for Ollama
Oh, you’re running from docker. You should have an .env.local file with the following specified for the Ollama base URL:
Your oTToDev instance is trying to locate Ollama at http://127.0.0.1:11434, which is not valid while you’re running within a Docker container based on this output in the video you posted (thanks for the video!)
Please update .env.local, rebuild based on docker instructions and then re-run the container. Hopefully that takes care of it.