llama_index: [Bug]: StructuredPlannerAgent always defaults to single task plan.

Bug Description

StructuredPlannerAgent doesn’t seem to be creating sub-tasks 😦

I tried my own toolset and wasn’t getting any success - my setup has multiple tools: txt2sql, visualization, semantic search, etc.

Then I thought maybe my use-case was too complex so I tried a dummy example with tools = [ product_price_tool, products_available_tool, product_description_tool, check_availability_tool ] and I asked things like “for each product available, get me the their prices and find me the description of the cheapest product”, etc… no sub-tasks.

Finally I gave up and tried the example from the documentation (see code below) because I thought maybe it didn’t find simple function calls ‘complex enough’ and it may work with query_index better… but still no sub-tasks (unlike what’s shown in the example)… the only difference between my code and the example in the doc is that I’m using VertexAI/Gemini instead of chatgpt4/openai

It simply never creates a multi sub-task plan, I even tried changing the initial_plan_prompt and plan_refine_prompt to emphasize always creating sub-tasks no matter the complexity, etc… but no success

Btw, I do get the answers in the end but it behaves as if I just used a FunctionCallingAgent not as the StructuredPlannerAgent

Version

0.11.18

Steps to Reproduce

from llama_index.core.tools import FunctionTool
        from llama_index.core.agent import FunctionCallingAgentWorker, StructuredPlannerAgent
        from llama_index.llms.vertex import Vertex
        from llama_index.core import Settings
        from llama_index.embeddings.vertex import VertexTextEmbedding
        from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
        from llama_index.core.tools import QueryEngineTool

        embed_model = VertexTextEmbedding(
            client_email=...
            token_uri=...
            private_key=...
            private_key_id=...
        )

        llm = Vertex(model='gemini-1.5-pro-002',
                    max_tokens=8000,
                    temperature=0.1)

        Settings.embed_model = embed_model
        Settings.llm = llm

        # Load documents, create tools
        lyft_documents = SimpleDirectoryReader(
            input_files=["pipelines/lyft_2021.pdf"]
        ).load_data()
        uber_documents = SimpleDirectoryReader(
            input_files=["pipelines/uber_2021.pdf"]
        ).load_data()

        lyft_index = VectorStoreIndex.from_documents(lyft_documents)
        uber_index = VectorStoreIndex.from_documents(uber_documents)

        lyft_tool = QueryEngineTool.from_defaults(
            lyft_index.as_query_engine(llm = llm),
            name="lyft_2021",
            description="Useful for asking questions about Lyft's 2021 10-K filling.",
        )

        uber_tool = QueryEngineTool.from_defaults(
            uber_index.as_query_engine(llm = llm),
            name="uber_2021",
            description="Useful for asking questions about Uber's 2021 10-K filling.",
        )

        tools = [
            lyft_tool, 
            uber_tool, 
    
        ]

        agent_worker = FunctionCallingAgentWorker.from_tools(
            tools, llm=llm,
            verbose=True
        )


        agent = StructuredPlannerAgent(
            agent_worker, tools=tools, verbose=True, llm=llm,
        )


        message = "Summarize the key risk factors for Lyft and Uber in their 2021 10-K filings."

        plan_id = agent.create_plan(message)
        plan = agent.state.plan_dict[plan_id]

        yield f"\n{plan}"

        response = agent.chat(message)

        yield f"\n{response}"

Relevant Logs/Tracbacks

No complex plan predicted. Defaulting to a single task plan.
=== Initial plan ===
default:
Summarize the key risk factors for Lyft and Uber in their 2021 10-K filings. ->

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24

Commits related to this issue

Most upvoted comments

After conversation with Google support, it is because of the way that pydantic serializes the Plan object. The internal schema object for google-cloud-aiplatform doesn’t recognize the schema type $ref or $def so it throws this error. Pydantic does not provide a mechanism to rename the schema field - it provides a template value for the reference values but not the names Pydantic Documentation.

I’ve pointed out to Google that they need to add this support to their library because it isn’t possible to make this change to pydantic.

I didn’t have much luck with it - but I did get it to work; I also got the Planner agent in this defect to work. I’ll push that to my open pr and fork tonight if you want to pull it down and test it out to give me feedback.

This is now a Google issue you can look follow the status at the link and star it so you get updates as it progresses.

I’ll continue to plug away at custom serialization formats to make this work - I think I found a way; but only some experimentation will tell.

I haven’t looked at FunctionCallingAgent, but I don’t think it serializes a Plan object into it’s tool call - which is what serializes with the json schema in it’s tool_parameter field.

I’m trying to see if I can override how the to_gemini_tool method creates the tool objects and perform a google-compatible serialization but finding a generic serialization that is compatible with both protobuf and pydantic is challenging - and I’m not familiar enough with either to just whip one up. I’ll keep plugging away - both at google and at the problem to see if I can get this figured out and into a PR.

I’m working on this and other issues with Vertex/Gemini - the update to google’s generativeai 1.70.0 version breaks a lot of things in the integration and it is taking some time for me to get to them- but this is my day job for my company so I’m going to keep plugging away at it and eventually I’ll have a version of the integration that works with the current Gemini/Vertex versions.

This particular problem is because the google library itself uses an out of date internal dependency - I’ve reached out to my Google support rep to see if I can get that issue resolved (or at least get a hotfix for it) but I only just hit this issue and it’s late on a Friday so I don’t really expect any movement today.