langchain: Using GPT 4 or GPT 3.5 with SQL Database Agent throws OutputParserException: Could not parse LLM output:

System Info

python 3.10 ubuntu Ubuntu 22.04.2 LTS langchain 0.0.194

Who can help?

@eyurtsev

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

from langchain.agents.agent_toolkits import SQLDatabaseToolkit
from langchain.sql_database import SQLDatabase
from langchain.agents import create_sql_agent
from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI
import os

os.environ["OPENAI_API_KEY"] = ""
db = SQLDatabase.from_uri(
    "postgresql://<my-db-uri>",
    engine_args={
        "connect_args": {"sslmode": "require"},
    },
)

llm = ChatOpenAI(model_name="gpt-3.5-turbo")
toolkit = SQLDatabaseToolkit(db=db, llm=llm)

agent_executor = create_sql_agent(
    llm=llm,
    toolkit=toolkit,
    verbose=True,
)

agent_executor.run("list the tables in the db. Give the answer in a table json format.")

Expected behavior

I am using the SQL Database Agent to query a postgres database. I want to use gpt 4 or gpt 3.5 models in the OpenAI llm passed to the agent, but it says I must use ChatOpenAI. Using ChatOpenAI throws parsing errors.

The reason for wanting to switch models is reduced cost, better performance and most importantly - token limit. The max token size is 4k for ‘text-davinci-003’ and I need at least double that.

When I do, it throws an error in the chain midway saying

> Entering new AgentExecutor chain...
Traceback (most recent call last):
  File "/home/ramlah/Documents/projects/langchain-test/sql.py", line 96, in <module>
    agent_executor.run("list the tables in the db. Give the answer in a table json format.")
  File "/home/ramlah/Documents/projects/langchain/langchain/chains/base.py", line 236, in run
    return self(args[0], callbacks=callbacks)[self.output_keys[0]]
  File "/home/ramlah/Documents/projects/langchain/langchain/chains/base.py", line 140, in __call__
    raise e
  File "/home/ramlah/Documents/projects/langchain/langchain/chains/base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/home/ramlah/Documents/projects/langchain/langchain/agents/agent.py", line 953, in _call
    next_step_output = self._take_next_step(
  File "/home/ramlah/Documents/projects/langchain/langchain/agents/agent.py", line 773, in _take_next_step
    raise e
  File "/home/ramlah/Documents/projects/langchain/langchain/agents/agent.py", line 762, in _take_next_step
    output = self.agent.plan(
  File "/home/ramlah/Documents/projects/langchain/langchain/agents/agent.py", line 444, in plan
    return self.output_parser.parse(full_output)
  File "/home/ramlah/Documents/projects/langchain/langchain/agents/mrkl/output_parser.py", line 51, in parse
    raise OutputParserException(
langchain.schema.OutputParserException: Could not parse LLM output: `Action: list_tables_sql_db, ''`

If I change the model to gpt-4, it runs one step then throws the error on the Thought for the next step

> Entering new AgentExecutor chain...
Action: list_tables_sql_db
Action Input: 
Observation: users, organizations, plans, workspace_members, curated_topic_details, subscription_modifiers, workspace_member_roles, receipts, workspaces, domain_information, alembic_version, blog_post, subscriptions
Thought:I need to check the schema of the blog_post table to find the relevant columns for social interactions.
Action: schema_sql_db
Action Input: blog_post
Observation: 
CREATE TABLE blog_post (
        id UUID NOT NULL, 
        category VARCHAR(255) NOT NULL, 
        title VARCHAR(255) NOT NULL, 
        slug VARCHAR(255) NOT NULL, 
        introduction TEXT NOT NULL, 
        list_of_blogs JSON[], 
        og_image VARCHAR(255), 
        created_at TIMESTAMP WITHOUT TIME ZONE NOT NULL, 
        updated_at TIMESTAMP WITHOUT TIME ZONE NOT NULL, 
        meta_description TEXT, 
        CONSTRAINT blog_post_pkey PRIMARY KEY (id)
)

/*
3 rows from blog_post table:
*** removing for privacy reasons ***
*/
Thought:Traceback (most recent call last):
  File "/home/ramlah/Documents/projects/langchain-test/sql.py", line 84, in <module>
    agent_executor.run("Give me the blog post that has the most social interactions.")
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 256, in run
    return self(args[0], callbacks=callbacks)[self.output_keys[0]]
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 145, in __call__
    raise e
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 139, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 953, in _call
    next_step_output = self._take_next_step(
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 773, in _take_next_step
    raise e
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 762, in _take_next_step
    output = self.agent.plan(
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 444, in plan
    return self.output_parser.parse(full_output)
  File "/home/ramlah/Documents/projects/langchain-test/venv/lib/python3.10/site-packages/langchain/agents/mrkl/output_parser.py", line 42, in parse
    raise OutputParserException(
langchain.schema.OutputParserException: Could not parse LLM output: `The blog_post table has a column list_of_blogs which seems to contain the social interaction data. I will now order the rows by the sum of their facebook_shares and twitter_shares and limit the result to 1 to get the blog post with the most social interactions.`

The error is inconsistent and sometimes the script runs normally.

• I have tried removing and adding streaming=True thinking that might be the cause.
• I have tried changing the model from gpt-3.5-turbo to gpt-4 as well, the error shows up inconsistently

Please let me know if I can provide any further information. Thanks!