langchainjs: The ChatOpenAI message parser does not handle OpenAI sending "content" along with any function calls

@jacoblee93, it seems that we have a more systemic issue here that requires your expertise to figure out how all these components should work together and if we can account for the original problem.

🤖

Thank you for bringing this to our attention. It seems like the parser is currently designed to handle either content or a function_call, but not both at the same time. To modify the parser to handle both, you would need to adjust the parsing logic in the output_parser.ts file.

Here’s a rough idea of how you could modify the parse method to handle both content and function_call:

parse(output: LLMResult): ParsedOutput {
  const parsedOutput: ParsedOutput = super.parse(output);
  const message = output.choices[0].message;

  if (message.function_call) {
    parsedOutput.function_call = message.function_call;
  }

  if (message.content) {
    parsedOutput.content = message.content;
  }

  return parsedOutput;
}

In this code, we first call the super.parse(output) method to get the initial parsed output. Then, we check if the message includes a function_call or content and add them to the parsed output if they exist.

Please note that this is a simplified example and you might need to adjust it to fit your specific needs. Also, remember to update the ParsedOutput interface to include the content field:

interface ParsedOutput {
  function_call?: FunctionCall;
  content?: string;
  // other fields...
}

This change should allow the parser to handle cases where OpenAI responds with both content and a function call. Please let me know if you have any questions or need further clarification.

Sources

• examples/src/prompts/structured_parser_zod.ts
• examples/src/prompts/structured_parser.ts
• langchain/src/schema/output_parser.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Thanks for the help @jacoblee93. I’ll update the thread on what I ended up doing just for completeness in case anyone else has a similar issue.

🤖

Based on your request, you can indeed implement a wrapper for the OpenAI model in the LangChain JS framework to intercept responses and split them into separate messages when both content and function_call are present in the same message. This can be achieved by modifying the openAIResponseToChatMessage function and _generate function in the ChatOpenAI class.

Here’s how you can do it:

1. Modify the openAIResponseToChatMessage function to return an array of messages instead of a single message when both content and function_call are present:

function openAIResponseToChatMessage(
  message: OpenAIClient.Chat.Completions.ChatCompletionMessage
): BaseMessage[] {
  if (message.role === "assistant" && message.content && message.function_call) {
    // If both content and function_call are present, split them into separate messages
    return [
      new AIMessage(message.content, {}),
      new AIMessage("", {
        function_call: message.function_call,
        tool_calls: message.tool_calls,
      }),
    ];
  } else if (message.role === "assistant") {
    return [
      new AIMessage(message.content || "", {
        function_call: message.function_call,
        tool_calls: message.tool_calls,
      }),
    ];
  } else {
    return [new ChatMessage(message.content || "", message.role ?? "unknown")];
  }
}

1. Modify the _generate function in the ChatOpenAI class to handle the array of messages returned by openAIResponseToChatMessage:

for (const part of data?.choices ?? []) {
  const messages = openAIResponseToChatMessage(
    part.message ?? { role: "assistant" }
  );
  for (const message of messages) {
    const text = message.content;
    const generation: ChatGeneration = {
      text,
      message: message,
    };
    if (part.finish_reason) {
      generation.generationInfo = { finish_reason: part.finish_reason };
    }
    generations.push(generation);
  }
}

This way, when both content and function_call are present in the same message, they will be split into separate messages. This should allow the parse method and the AgentExecutor to handle each message correctly.

You can find the relevant code in the langchain/src/chat_models/openai.ts file in the langchainjs repository.

Please note that this modification should be done carefully to avoid any unintended side effects. Make sure to thoroughly test your application after implementing these changes to ensure everything works as expected.

Sources

• langchain/src/chat_models/openai.ts
• langchain/src/llms/openai-chat.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

🤖

Based on the information you’ve provided and the context from the repository, it seems like you’re encountering an issue where the content field is being ignored when both content and a function_call are present in the OpenAI response.

A similar issue was previously encountered and resolved in the LangChain JS repository. The issue was titled Langchain JS PromptLayerChatOpenAI doesnt recognize message with type == “function”. The solution involved modifying the messageTypeToOpenAIRole and openAIResponseToChatMessage functions to handle the “function” message type. This change was implemented in pull request #1948.

However, you’ve mentioned that you’re looking for a solution that works with the agent executor without modifying the internals. The AgentExecutor class in the executor.ts file handles the output of the parse method through its _call method. If an OutputParserException is thrown during the execution of the agent’s plan, the AgentExecutor handles it based on the handleParsingErrors property. This could potentially be a place to handle the issue you’re facing.

Unfortunately, without modifying the internals of the LangChain JS framework, it might be challenging to achieve the desired behavior. The framework’s design seems to assume that a message will either contain content or a function_call, but not both.

If you have control over the OpenAI responses, one workaround could be to ensure that the content and function_call are not included in the same message. Instead, you could split them into two separate messages, each with its own role. This way, the parse method and the AgentExecutor should be able to handle each message correctly.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• Langchain JS PromptLayerChatOpenAI doesnt recognize message with type == “function”
• langchain/src/agents/executor.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.