llama_index: [Bug]: Not able to make call to Ollama multimodal model in the cookbook

Bug Description

When I trying to follow the steps int the cookbook, I was getting an error saying that there’s a ResponseError: calling Ollama. However, I have installed Ollama successfully and was able to make model call via curl command. I suspect that this has something to do with http://localhost:11434.

Version

llama-index 0.9.44

Steps to Reproduce

Following the steps in the cookbook

from pydantic import BaseModel

class Restaurant(BaseModel):
    """Data model for an restaurant."""

    restaurant: str
    food: str
    discount: str
    price: str
    rating: str
    review: str

from llama_index.program import MultiModalLLMCompletionProgram
from llama_index.output_parsers import PydanticOutputParser

prompt_template_str = """\
{query_str}

Return the answer as a Pydantic object. The Pydantic schema is given below:

"""
mm_program = MultiModalLLMCompletionProgram.from_defaults(
    output_parser=PydanticOutputParser(Restaurant),
    image_documents=image_documents,
    prompt_template_str=prompt_template_str,
    multi_modal_llm=mm_model,
    verbose=True,
)

response = mm_program(query_str="Can you summarize what is in the image?")
for res in response:
    print(res)

Relevant Logs/Tracbacks

{ “name”: “ResponseError”, “message”: “”, “stack”: "--------------------------------------------------------------------------- ResponseError Traceback (most recent call last) Cell In[13], line 1 ----> 1 response = mm_program(query_str="Can you summarize what is in the image?") 2 for res in response: 3 print(res)

File /usr/local/lib/python3.11/site-packages/llama_index/program/multi_modal_llm_program.py:85, in MultiModalLLMCompletionProgram.call(self, llm_kwargs, *args, **kwargs) 82 llm_kwargs = llm_kwargs or {} 83 formatted_prompt = self._prompt.format(llm=self._multi_modal_llm, **kwargs) —> 85 response = self._multi_modal_llm.complete( 86 formatted_prompt, 87 image_documents=self._image_documents, 88 **llm_kwargs, 89 ) 91 raw_output = response.text 92 if self._verbose:

File /usr/local/lib/python3.11/site-packages/llama_index/multi_modal_llms/ollama.py:158, in OllamaMultiModal.complete(self, prompt, image_documents, formatted, **kwargs) 155 """Complete.""" 156 import ollama –> 158 response = ollama.generate( 159 model=self.model, 160 prompt=prompt, 161 images=image_documents_to_base64(image_documents), 162 stream=False, 163 options=self._model_kwargs, 164 ) 165 return CompletionResponse( 166 text=response["response"], 167 raw=response, 168 additional_kwargs=get_additional_kwargs(response, ("response",)), 169 )

File /usr/local/lib/python3.11/site-packages/ollama/_client.py:111, in Client.generate(self, model, prompt, system, template, context, stream, raw, format, images, options, keep_alive) 108 if not model: 109 raise RequestError(‘must provide a model’) –> 111 return self._request_stream( 112 ‘POST’, 113 ‘/api/generate’, 114 json={ 115 ‘model’: model, 116 ‘prompt’: prompt, 117 ‘system’: system, 118 ‘template’: template, 119 ‘context’: context or [], 120 ‘stream’: stream, 121 ‘raw’: raw, 122 ‘images’: [_encode_image(image) for image in images or []], 123 ‘format’: format, 124 ‘options’: options or {}, 125 ‘keep_alive’: keep_alive, 126 }, 127 stream=stream, 128 )

File /usr/local/lib/python3.11/site-packages/ollama/_client.py:82, in Client._request_stream(self, stream, *args, **kwargs) 76 def _request_stream( 77 self, 78 *args, 79 stream: bool = False, 80 **kwargs, 81 ) -> Union[Mapping[str, Any], Iterator[Mapping[str, Any]]]: —> 82 return self._stream(*args, **kwargs) if stream else self._request(*args, **kwargs).json()

File /usr/local/lib/python3.11/site-packages/ollama/_client.py:58, in Client._request(self, method, url, **kwargs) 56 response.raise_for_status() 57 except httpx.HTTPStatusError as e: —> 58 raise ResponseError(e.response.text, e.response.status_code) from None 60 return response

ResponseError: " }