llama_index: [Bug]: Vertex LLM does not Handle FunctionCall tools

Bug Description

When using a FunctionTool, I receive the following error: AttributeError: 'FunctionCall' object has no attribute '_pb'

It seems that this tool makes some assumptions about the input data types.

Version

0.11.7, llama-index-llms-vertex==0.3.7

Steps to Reproduce

Update the code in The multi-agent concierge example to use Vertex, then try and run.

Relevant Logs/Tracbacks

INFO:chromadb.telemetry.product.posthog:Anonymized telemetry enabled. See                     https://docs.trychroma.com/telemetry for more information.
ERROR:asyncio:Exception in callback Dispatcher.span.<locals>.wrapper.<locals>.handle_future_result(span_id='Workflow.run...-6419e73b17c5', bound_args=<BoundArgumen...bose': True})>, instance=<clgx.agent.e...t 0x3230e6a50>, context=<_contextvars...t 0x31fb73e00>)(<WorkflowHand...ibute '_pb'")>) at /Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py:273
handle: <Handle Dispatcher.span.<locals>.wrapper.<locals>.handle_future_result(span_id='Workflow.run...-6419e73b17c5', bound_args=<BoundArgumen...bose': True})>, instance=<clgx.agent.e...t 0x3230e6a50>, context=<_contextvars...t 0x31fb73e00>)(<WorkflowHand...ibute '_pb'")>) at /Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py:273>
Traceback (most recent call last):
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 283, in handle_future_result
    raise exception
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/CoreSearchAgent.py", line 242, in main
    result = await handler
             ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 290, in __await__
    return self.result()  # May raise too.
           ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 376, in _run_workflow
    raise exception_raised
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 233, in _task
    raise e from None
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 229, in _task
    new_ev = await instrumented_step(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 357, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/events/CoreSearchEvents.py", line 358, in orchestrator
    tool_calls = llm.get_tool_calls_from_response(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/llms/vertex/base.py", line 498, in get_tool_calls_from_response
    response_dict = MessageToDict(tool_call._pb)
                                  ^^^^^^^^^^^^^
AttributeError: 'FunctionCall' object has no attribute '_pb'
Traceback (most recent call last):
  File "/Users/stfines/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers-pro/pydevd_asyncio/pydevd_nest_asyncio.py", line 138, in run
    return loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers-pro/pydevd_asyncio/pydevd_nest_asyncio.py", line 243, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/CoreSearchAgent.py", line 242, in main
    result = await handler
             ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 290, in __await__
    return self.result()  # May raise too.
           ^^^^^^^^^^^^^
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 376, in _run_workflow
    raise exception_raised
  File "/Users/stfines/.pyenv/versions/3.11.6/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 233, in _task
    raise e from None
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/workflow/workflow.py", line 229, in _task
    new_ev = await instrumented_step(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 357, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/workspace/propertyPoc-LlamaIndex/clgx/agent/events/CoreSearchEvents.py", line 358, in orchestrator
    tool_calls = llm.get_tool_calls_from_response(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stfines/Library/Caches/pypoetry/virtualenvs/propertypoc-llamaindex-CWmV6wNU-py3.11/lib/python3.11/site-packages/llama_index/llms/vertex/base.py", line 498, in get_tool_calls_from_response
    response_dict = MessageToDict(tool_call._pb)
                                  ^^^^^^^^^^^^^
AttributeError: 'FunctionCall' object has no attribute '_pb'

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 17 (3 by maintainers)

Commits related to this issue

Most upvoted comments

I can confirm that this workaround work as well with the latest version of the google-cloud-aiplatform. I did not see any other issues so far except from the fact that llamaindex will always try to call .text on the response (https://github.com/run-llama/llama_index/blob/v0.12.15/llama-index-integrations/llms/llama-index-llms-vertex/llama_index/llms/vertex/base.py#L286 and https://github.com/run-llama/llama_index/blob/v0.12.15/llama-index-integrations/llms/llama-index-llms-vertex/llama_index/llms/vertex/base.py#L429 which fails if the model only generated a function call.

I will try to open a PR to bump google-cloud-aiplatform to 1.79.0 and include the fix from @jeffbryner as well as checking what I can do about the .text being call, however I’m a bit worried that it might break a lot of other stuff that used to work.

@stfines-clgx any reason why you kept this in a separate branch ? it seems that you changed a lot more than only fixing some bugs, is there any chance you could include the fix as well and use this as a base for the PR to upgrade Vertex AI ?

I did not worked on that as we ended up using Gemini api instead of vertex, we however had to fix a similar issue in the Gemini api (llama-index-llms-gemini 0.4.6+) that we could probably draw inspiration from. I will not have the time to work on the vertex api as this is not a priority anymore

Ok this fork has a branch - dev/1_70_protobuf_only that seems to resolve this issue. My testing with several different types of functions and non-function invocations has it working well. It does not resolve #16625 - it looks like that will take action from the google-cloud-aiplatform team though I will continue to explore mitigations.

@logan-markewich if you would take a look prior to my creating a PR and let me know what else needs doing to make a good PR for this project I would appreciate it.

That does seem possible, esp since the pyproject.toml lists a aiplatform dependency of 1.39.0 and the current version is 1.70.0; I’ll pull locally and see if I can build this lib with a newer dependency