cody: bug: cannot configure `unstable-ollama` provider, ignores model and Server Endpoint settings

Version

v1.2.3

Describe the bug

When selecting to use a self hosted ollama instance, there is no way to do 2 things:

  1. Set the server endpoint for the ollama instance. in my case I have a desktop machine with a good GPU and run ollama there, when coding on my laptop i want to use the ollama instance on my desktop, no matter what value is set for cody.autocomplete.advanced.serverEndpoint, cody will always attempt to use http://localhost:11434, so i cannot sepcify the ip of my desktop machine hosting ollama.
  2. Use a different model on ollama - no matter what value is set for cody.autocomplete.advanced.model, for example when llama-code-13b is selected, the vscode output tab for cody always says: █ CodyCompletionProvider:initialized: unstable-ollama/codellama:7b-code

Expected behavior

When a esrver endpoint is specified, the ollama configuration should use that value instead of the default http://localhost:11434 when a model is selected it should use that model in the config and request paramteres, instead of always using codellama:7b-code

Additional context

image image shows output for unstable-ollama/codellama:7b-code being used despite 13b being selected.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 21 (6 by maintainers)

Most upvoted comments

@lukeab cody.autocomplete.advanced.serverEndpoint is actually not used when the Ollama provider is configured. Sorry for that (I know it’s super confusing and we have to make this easier, sorry!). You want to use the cody.autocomplete.experimental.ollamaOptions option like this:

  "cody.autocomplete.advanced.provider": "experimental-ollama",
  "cody.autocomplete.experimental.ollamaOptions": {
    "url": "http://localhost:11434",
    "model": "deepseek-coder:6.7b-base-q4_K_M"
  },

@lakarpusky Which doc do you refer to? We have renamed the option from unstable-ollama to experimental-ollama, so that doc will need updating I’m afraid.

Is there a plan to enable Ollama for chat?

@lakarpusky Unfortunlaty we don’t have support for Ollama in Chat yet. 😕

+1

The document is old.

I am trying to configure Cody with local ollama, the doc says to select the unstable-ollama provider but it does not appear in the list.

VSCode v1.85.2 Cody AI v1.4.4

CleanShot 2024-02-18 at 22 45 03

CleanShot 2024-02-18 at 22 39 02

@jdorfman @philipp-spiess For me these options in VS Code 1.90.0 with Cody 1.20.3 no longer seem to work:

    "cody.autocomplete.advanced.provider": "experimental-ollama",
    "cody.autocomplete.experimental.ollamaOptions": {
        "url": "http://localhost:11434",
        "model": "granite-code:34b"
    },
    "cody.autocomplete.languages": {
        "*": true
    },
    "cody.experimental.ollamaChat": true,

Namely, the cody.autocomplete.experimental.ollamaOptions and cody.experimental.ollamaChat show the Unknown Configuration Setting tooltip.

If I click Cody: Generate Code option from lightbulb icon I get No chat model found in server-provided config and output window is showing:

ConfigFeaturesSingleton: refreshConfigFeatures accessing Sourcegraph GraphQL API: Error: HTTP status code 401: Invalid access token.

Note that I am not logged into sourcegraph account, but I am also not getting any auto-completion offers.

Another error I am seeing in earlier output is:

LocalEmbeddingsController: load 5f1385928dff462ab5e748800ac37144 {"code":-32099}

Could you please point me to some updated docs which say what exactly is supported in current version and what are the correct configuration keys now?

@lukeab cody.autocomplete.advanced.serverEndpoint is actually not used when the Ollama provider is configured. Sorry for that (I know it’s super confusing and we have to make this easier, sorry!). You want to use the cody.autocomplete.experimental.ollamaOptions option like this:

  "cody.autocomplete.advanced.provider": "experimental-ollama",
  "cody.autocomplete.experimental.ollamaOptions": {
    "url": "http://localhost:11434",
    "model": "deepseek-coder:6.7b-base-q4_K_M"
  },

@lakarpusky Which doc do you refer to? We have renamed the option from unstable-ollama to experimental-ollama, so that doc will need updating I’m afraid.

It is not clear to me how I can add the following to my visual code settings file

"cody.autocomplete.experimental.ollamaOptions": {
  "url": "http://localhost:11434",
  "model": "deepseek-coder:6.7b-base-q4_K_M"
},