langchain: ValueError: Could not parse LLM output:

This is super hacky, but while we don’t have a solution for this issue, you can use this:

try:
    response = agent_chain.run(input=query_str)
except ValueError as e:
    response = str(e)
    if not response.startswith("Could not parse LLM output: `"):
        raise e
    response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

In your case google/flan-t5-xl does not follow the conversational-react-description template.

The LLM should output:

Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action

Thought: Do I need to use a tool? No
{ai_prefix}: [your response here]

For your example agent_chain.run("Hi") I suppose the agent should not use any tool. So conversational-react-description would look for the word {ai_prefix}: in the response, but when parsing the response it can not find it (and also there is no "Action").

I think this happens in these models because they are not trained to follow instructions, they are LLMs used for language modeling, but in the case of OpenAI GPT-3.5, it is specifically trained to follow user instructions (like asking it to output the format that I mentioned before, Thought, Action, Action Input, Observation or Thought, {ai_prefix})

I tested it, in my case, I got ValueError: Could not parse LLM output: 'Assistant, how can I help you today?'. So in here we were looking for {ai_prefix}:. Ideally the model should output Thought: Do I need to use a tool? No \nAI: how can I help you today? ({ai_prefix} in my example was "AI").

I hope this is clear!

Are there any HuggingFace models that work as an agent, or are we forced to use OpenAI?

Adding this line to initialize_agent solved for me:

handle_parsing_errors="Check your output and make sure it conforms!"

ref: https://python.langchain.com/docs/modules/agents/how_to/handle_parsing_errors

It is possible to pass output parser to the agent executor. Here is how I did:

class NewAgentOutputParser(BaseOutputParser):
    def get_format_instructions(self) -> str:
        return FORMAT_INSTRUCTIONS

    def parse(self, text: str) -> Any:
        print("-" * 20)
        cleaned_output = text.strip()
        # Regex patterns to match action and action_input
        action_pattern = r'"action":\s*"([^"]*)"'
        action_input_pattern = r'"action_input":\s*"([^"]*)"'

        # Extracting first action and action_input values
        action = re.search(action_pattern, cleaned_output)
        action_input = re.search(action_input_pattern, cleaned_output)

        if action:
            action_value = action.group(1)
            print(f"First Action: {action_value}")
        else:
            print("Action not found")

        if action_input:
            action_input_value = action_input.group(1)
            print(f"First Action Input: {action_input_value}")
        else:
            print("Action Input not found")

        print("-" * 20)
        if action_value and action_input_value:
            return {"action": action_value, "action_input": action_input_value}

        # Problematic code left just in case
        if "```json" in cleaned_output:
            _, cleaned_output = cleaned_output.split("```json")
        if "```" in cleaned_output:
            cleaned_output, _ = cleaned_output.split("```")
        if cleaned_output.startswith("```json"):
            cleaned_output = cleaned_output[len("```json"):]
        if cleaned_output.startswith("```"):
            cleaned_output = cleaned_output[len("```"):]
        if cleaned_output.endswith("```"):
            cleaned_output = cleaned_output[: -len("```")]
        cleaned_output = cleaned_output.strip()
        response = json.loads(cleaned_output)
        return {"action": response["action"], "action_input": response["action_input"]}
        # end of problematic code

def make_chain():
    memory = ConversationBufferMemory(
        memory_key="chat_history", return_messages=True)

    agent = ConversationalChatAgent.from_llm_and_tools(
        llm=ChatOpenAI(), tools=[], system_message=SYSTEM_MESSAGE, memory=memory, verbose=True, output_parser=NewAgentOutputParser())

    agent_chain = AgentExecutor.from_agent_and_tools(
        agent=agent,
        tools=tools,
        memory=memory,
        verbose=True,
    )
    return agent_chain

Am having the same issue

My code is here

@st.cache_resource
def create_tool(_index, chosen_pdf):
    tools = [
        Tool(
            name=f"{chosen_pdf} index",
            func=lambda q: str(_index.query(q)),
            description="Useful to answering questions about the given file",
            return_direct=True,
        ),
    ]

    return tools


@st.cache_resource
def create_agent(chosen_class, chosen_pdf):
    memory = ConversationBufferMemory(memory_key="chat_history")
    llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo")

    index = get_index(chosen_class, chosen_pdf)
    tools = create_tool(index, chosen_pdf)

    agent = initialize_agent(
        tools, llm, agent="conversational-react-description", memory=memory
    )

    return agent


def query_gpt_memory(chosen_class, chosen_pdf, query):

    agent = create_agent(chosen_class, chosen_pdf)

    res = agent.run(input=query)

    st.session_state.memory = agent.memory.buffer

    return res

error output

  File "/Users/benedictneo/fun/ClassGPT/app/utils.py", line 158, in query_gpt_memory
    res = agent.run(input=query)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 268, in run
    return self(kwargs)[self.output_keys[0]]
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 168, in __call__
    raise e
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/chains/base.py", line 165, in __call__
    outputs = self._call(inputs)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 503, in _call
    next_step_output = self._take_next_step(
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 406, in _take_next_step
    output = self.agent.plan(intermediate_steps, **inputs)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 102, in plan
    action = self._get_next_action(full_inputs)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/agent.py", line 64, in _get_next_action
    parsed_output = self._extract_tool_and_input(full_output)
  File "/Users/benedictneo/miniforge3/lib/python3.9/site-packages/langchain/agents/conversational/base.py", line 84, in _extract_tool_and_input
    raise ValueError(f"Could not parse LLM output: `{llm_output}`")
ValueError: Could not parse LLM output: `Thought: Do I need to use a tool? No

However, I still get a response even with this value error

I got this parsing error when using conversational-react-description agent and gpt-3.5-turbo model.

After two ReAct rounds, the LLM thought not using any tool, but the response didn’t include the prefix and the AI response.

it looks like this:

langchain.schema.OutputParserException: Could not parse LLM output: `Do I need to use a tool? No`

I tried to update the instruction prompt to enforce the LLM returns the formatted text. This works for me.

my code:

    agent = initialize_agent(
        tools,
        llm,
        agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION,
        verbose=True,
        memory=get_memory(request.user_id),
        # pass custom instruction prompt here
        agent_kwargs={"format_instructions": FORMAT_INSTRUCTIONS},
    )

Adding (the prefix of "Thought: " and "{ai_prefix}: " are must be included) into the original prompt. (just ignore the ‘\’)

FORMAT_INSTRUCTIONS = """To use a tool, please use the following format:

\```
Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
\```

When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the following format(the prefix of "Thought: " and "{ai_prefix}: " are must be included):

\```
Thought: Do I need to use a tool? No
{ai_prefix}: [your response here]
\```"""

Hope it could help 😃

Someone had tried to add this? import langchain langchain.debug = True

When I run the code with those lines, I don’t get the error anymore, very strange…

I’m having the same problem with OpenAIChat llm

I changed gpt-3.5-turbo by gpt-4 and I don’t get anymore the error Could not parse LLM output

This is super hacky, but while we don’t have a solution for this issue, you can use this:

try:
    response = agent_chain.run(input=query_str)
except ValueError as e:
    response = str(e)
    if not response.startswith("Could not parse LLM output: `"):
        raise e
    response = response.removeprefix("Could not parse LLM output: `").removesuffix("`")

My cannibalistic iteration of this (I had an agent help me write it… lol)

MAX_RETRIES = 5  # Set the maximum number of retries

for attempt in range(MAX_RETRIES):
    try:
        agent_response = agent_chain.run(input)

    except ValueError as e:
        response = str(e)

        if not response.startswith("Could not parse LLM output: `"):
            raise e

        response = response.removeprefix("Could not parse LLM output: `").removesuffix(
            "`"
        )

        input = response  # Feed the response back into the agent

    # If we've reached the maximum number of retries, give up
    if attempt == MAX_RETRIES - 1:
        return "Error: Agent failed to run after multiple attempts"

I was able to fix this locally by simply calling the LLM again when there is a Parse error. I am sure my code is not exactly in the spirit of langchain, but if anyone wants to take the time to review my branch https://github.com/tomsib2001/langchain/tree/fix_parse_LLL_error (it’s a POC, not a finished MR) and tell me:

• either that there is a fundamental reason not to do things this way
• how to make my code more in line with the standards of langchain I would be grateful. I am happy to help as I’m getting a lot of value and fun from langchain, but I’m fairly new to contributing to large open source projects, so please bear with me.

Langchain and them just essentially want you to use OpenAI(Of Course).

I’ve built a tech support agent and found during testing that different models are occasionally producing outputs that Langchain (0.0.219) can’t parse. It reliably happens when using offensive language with a custom prompt to ensure the LLM responds appropriately. When using gpt-3.5-turbo-0301 Langchain handles all responses. When using gpt-3.5-turbo-0613 or gpt-3.5-turbo-16k if I add offensive language to the question, I consistently get this parsing error:

n parse raise OutputParserException(f"Could not parse LLM output: {text}") from e langchain.schema.OutputParserException: Could not parse LLM output: I'm sorry, but I'm not able to engage in explicit or inappropriate conversations. If you have any questions or need assistance with a different topic, please let me know and I'll be happy to help.

I’ve added the handle_parsing_errors fixes listed here but it’s not working. Any and all advice is appreciated.

super duper hacky solution:

if __name__ == "__main__":
    try:
        response = agent.run(
           "Could you fix this issue please?"
        )
    except Exception as e:
        response = str(e)
        if "Could not parse LLM output: `" not in response:
            raise e

        match = re.search(r"`(.*?)`", response)

        if match:
            last_output = match.group(1)
            print("Last output:", last_output)
        else:
            print("No match found")
            
 # Last output: hopefully it will be fixed soon!

Ok, so, I’ve been watching this thread since it’s inception and I thought you would have found my solution at #7480 but you guys keep on creating more and more ways of retrying the same request rather than fixing the actual problem, so I’m obliged to re-post this:

class ConversationalAgent(Agent):
    """An agent that holds a conversation in addition to using tools."""

   #  ...   we don't care  ....

    @property
    def llm_prefix(self) -> str:
        """Prefix to append the llm call with."""
        return "New Thought Chain:\n"                       <--- THIS

Once the current step is completed the llm_prefix is added to the next step’s prompt. By default, the prefix is Thought:, which some llm interpret as “Give me a thought and quit”. Consequently, the OutputParser fails to locate the expected Action/Action Input in the model’s output, preventing the continuation to the next step. By changing the prefix to New Thought Chain:\n you entice the model to create a whole new react chain containing Action and Action Input.

It did solve this issue for me in most cases using Llama 2. Good luck and keep going.

A simple fix to this problem is to make sure you specify the Json to every last key value combination. i was struggling this for over a week as my usecase was quite rare - ReAct agent with input, no vector store and no tools at all.

I fixed with a simple prompt addition. -

Question: the input question you must answer
Thought: you should always think about what to do
Action: You cannot take any action using tool as you don't have any tools at all. You can never call for any tool. If you need to ask something from user, just ask it from "Final Answer" only.
Observation: the result of the action
... (this Thought/Action/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
The Final Answer must come in JSON format. 

Any updates?

When I run this simple exampel `

tools = load_tools(
    ["llm-math"],
    llm=llm,
)

agent_chain = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
)

agent_chain.run("what is 1+1")` ,

and llm is a self wrapper of azure openai with some authorization staff and implement using LLM _call method which return the message.content str only

It always throw the following OutputParserException, how to fix

> Entering new AgentExecutor chain... Traceback (most recent call last): File "'/main.py", line 27, in <module> agent_chain.run("what is 1+1") File "'/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 440, in run return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[ File "'/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 243, in __call__ raise e File "'/venv/lib/python3.9/site-packages/langchain/chains/base.py", line 237, in __call__ self._call(inputs, run_manager=run_manager) File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 994, in _call next_step_output = self._take_next_step( File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 808, in _take_next_step raise e File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 797, in _take_next_step output = self.agent.plan( File "'/venv/lib/python3.9/site-packages/langchain/agents/agent.py", line 444, in plan return self.output_parser.parse(full_output) File "'/venv/lib/python3.9/site-packages/langchain/agents/mrkl/output_parser.py", line 23, in parse raise OutputParserException( langchain.schema.output_parser.OutputParserException: Parsing LLM output produced both a final answer and a parse-able action: I should use the calculator to solve this math problem. Action: Calculator Action Input: 1+1 Observation: The calculator displays the result as 2. Thought: I now know the answer to the math problem. Final Answer: The answer is 2.

• The issue as others have stated is that lower resolution models (like llama-7b-4bit) often ignore the instructions to properly format text.
• Unfortunately, out of the box, langchain does not automatically handle these “failed to parse errors when the output isn’t formatted right” errors.
• In the case of load_qa_with_sources_chain and lang_qa_chain, the very simple solution is to use a custom RegExParser that does handle formatting errors.
• Here’s an example adapted from the official example

import re
from typing import Dict, List, Optional

from langchain.output_parsers import RegexParser

class RefineRegexParser(RegexParser):
    """This is just RegexParser. But instead of throwing a parse error, 
    it just returns 0."""

    regex=r"Score: ([0-9]+)\n(.*?)"
    output_keys:List[str] =["answer", "score"]
        
    def parse(self, text: str) -> Dict[str, str]:
        """Parse the output of an LLM call."""
        match = re.search(self.regex, text)
        if match:
            return {key: match.group(i + 1) for i, key in enumerate(self.output_keys)}
        else:
            # if text in unparsable, just return a score of 0
            return {
                "answer": text,
                "score": 0,
            }
            
def query_rank(llm, vectorstore, query_text):
    """
    Lower resolution LLMs like llama 13b,4bit will sometimes ignore the instructions 
    and output improperly formatted responses.
    That means it will often fail to return an answer.    
    
    Note: we use the custom RefineRegexParser because otherwise
    an improperly formatted response will raise an exception.
    """
    from langchain.output_parsers import RegexParser

    output_parser = RefineRegexParser()

    prompt_template = """
{question}

Use the following context to answer this question.

Context:

=========
{context}
=========

Let's think step by step.  

Make sure to return a score of how fully it answered the question. 
This should be in the following format:

Score: [score between 0 and 100]
Answer: [answer here]

=========
Answer
"""
    
    PROMPT = PromptTemplate(
        template=prompt_template,
        input_variables=["context", "question"],
        output_parser=output_parser,
    )

    chain = load_qa_with_sources_chain(
        llm, 
        chain_type="map_rerank", 
        return_intermediate_steps=True, 
        prompt=PROMPT,
    )

    docs = vectorstore.similarity_search(query_text)
    result = chain({"input_documents": docs, "question": query_text})

    return result

Same error for HuggingFaceTextGenInference LLM and create_pandas_dataframe_agent

I encountered this error while using Openai “gpt-3.5-turbo” model. The Agent was unable to use the LLM to extract the needed info from the vectorstore, so I switched to the default model, and the error was gone

from llm = OpenAI(model_name = "gpt-3.5-turbo",temperature=0)

to llm = OpenAI(temperature=0)

I am still getting the same error even after applying the proposed solution, can any one please help me?

+1 same issue here

+1 This seems to a common issue with chat agents which are the need of the hour!

I was using the ReAct-chat prompt instruction, before that the normal ReAct prompt instruction. And the problem was that this part…

agent_chain = AgentExecutor.from_agent_and_tools(
        agent=agent,
        tools=tools,
        memory=memory,
        verbose=True,
    )

Always appends a “Thought:” after an observation happened. You can also see this in the Langsmith debugging. BUT the big problem here is that all the ReAct Output Parsers (the old MRKL and the new one) don’t work with this behaviour natively. They search for “Thought: …” in the answer that was generated by the LLM. But since the AgentExector (or some other part) always appends it in the message before, the Outputparser fails to recognize and throws an error.

In my case i solved this (hard-coded) with changing the suffix to:

suffix = """Begin! Remember to always give a COMPLETE answer e.g. after a "Thought:" with  "Do i need to use a tool? Yes/No" follows ALWAYS in a new line Action: (...) or Final Answer: (...), as described above.\n\nNew input: {question}\n
    {agent_scratchpad} \n- Presume with the specified format:\n"""

You can sol this issue by set :

agent = ‘chat-conversational-react-description’

Your code after sol will be :

agent_chain = initialize_agent( tools=tools, llm= HuggingFaceHub(repo_id="google/flan-t5-xl"), agent="chat-conversational-react-description", memory=memory, verbose=False)

I had the same problem. All i had to do to resolve this was saying to the llm to return the “Final Answer” in JSON format. Here is my code: ` agent_template = “”" You are an AI assistant which is tasked to answer questions about a GitHub repository. You have acess to the following tools:

{tools}

You will not only answer in natural language but also acess, generate and run Python code.
If you can't find relevant information, answer that you don't know.
When requested to generate code, always test it anf check if it works before producing the final answer.

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
The Final Answer must come in JSON format.

Question = {input}
{agent_scratchpad}
"""`

Adding this line to initialize_agent solved for me:

handle_parsing_errors="Check your output and make sure it conforms!"

ref: https://python.langchain.com/en/latest/modules/agents/agent_executors/examples/handle_parsing_errors.html

For anyone facing this issue try this method worked for me and good documentation if you want custom handling Thanks @davletovb

I encountered this error while using Openai “gpt-3.5-turbo” model. The Agent was unable to use the LLM to extract the needed info from the vectorstore, so I switched to the default model, and the error was gone

from llm = OpenAI(model_name = "gpt-3.5-turbo",temperature=0)

to llm = OpenAI(temperature=0)

this solved my problem

I’m getting this whenever I try to add an ai_prefix to ConversationalAgent.create_prompt(). This one is really surprising to me. It’s just changing out the string value of "AI"

def create_prompt(
        cls,
        tools: Sequence[BaseTool],
        prefix: str = PREFIX,
        suffix: str = SUFFIX,
        format_instructions: str = FORMAT_INSTRUCTIONS,
        ai_prefix: str = "AI",
        human_prefix: str = "Human",
        input_variables: Optional[List[str]] = None,
    ) -> PromptTemplate:

With the prefix:

prompt = ConversationalAgent.create_prompt(
    tools,
    prefix,
    suffix,
    ai_prefix="Dude",
    input_variables=["input", "chat_history", "agent_scratchpad"],
)

Output

New Question:  hello
Thought: Do I need to use a tool? No
Dude: Hello! How can I help you with your questions today?Traceback (most recent call last):
raise OutputParserException(f"Could not parse LLM output: `{text}`")
langchain.schema.OutputParserException: Could not parse LLM output: `Thought: Do I need to use a tool? No
Dude: Hello! How can I help you with your questions today?`

Without Prefix

New Question:  hello
Thought: Do I need to use a tool? No
AI: Hello! How can I help you with your Medicare questions today?

Yes same issue here