llama_index: [Bug]: Using AzureOpenAI with OpenAIAgent

Bug Description

Previous issue was closed. so I repost my comment as a new issue:

I tried the following model in Azure -> [“gpt-35-turbo”, “gpt-35-turbo-16k”, “gpt-4”, “gpt-4-32k”] with api_version = “2023-07-01-preview”

All of them failed due to same reason: BadRequestError: Error code: 400 - {‘error’: {‘message’: ‘Unrecognized request arguments supplied: tool_choice, tools’, ‘type’: ‘invalid_request_error’, ‘param’: None, ‘code’: None}}

If you look into openai python sdk openai > resources > chat > completions > Completions > create the function_call and functions are actually deprecated in favor of tool_choice and tools

However, if you use AzureOpenAI,

functions = [
    {
        "name": "get_current_weather",
        "description": "Get the current weather",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "format": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "The temperature unit to use. Infer this from the users location.",
                },
            },
            "required": ["location"],
        },
    }
]

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "format": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use. Infer this from the users location.",
                    },
                },
                "required": ["location"],
            },
        },
    }
]

This works

messages = [
    {"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."},
    {"role": "user", "content": "What's the weather like today in Seattle?"}
]

chat_completion = client.chat.completions.create(
    model=deployment,
    messages=messages,
    functions=functions,
)
print(chat_completion)

This doesn’t:

messages = [
    {
        "role": "system",
        "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.",
    },
    {"role": "user", "content": "What's the weather like today in Seattle?"},
]

chat_completion = client.chat.completions.create(
    model=deployment,
    messages=messages,
    tools=tools,
)
print(chat_completion)

So it seems like the Azure models have not yet accepted tools and tool_choice params. https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/function-calling?tabs=python-new

Version

0.9.5

Steps to Reproduce

from llama_index.tools import FunctionTool
from llama_index.llms import AzureOpenAI
from llama_index.embeddings import AzureOpenAIEmbedding

api_key = "<your-api-key>"
azure_endpoint = "https://<your-endpoint>.openai.azure.com/"
api_version = "2023-07-01-preview"

llm = AzureOpenAI(
    model="gpt-4",
    deployment_name="<your-deployment>",
    api_key=api_key,
    azure_endpoint=azure_endpoint,
    api_version=api_version,
)

def multiply(a: int, b: int) -> int:
    """Multiply two integers and returns the result integer"""
    return a * b


def add(a: int, b: int) -> int:
    """Add two integers and returns the result integer"""
    return a + b


multiply_tool = FunctionTool.from_defaults(fn=multiply, name="multiply")

add_tool = FunctionTool.from_defaults(fn=add, name="add")

all_tools = [multiply_tool, add_tool]

agent = OpenAIAgent.from_tools(all_tools, llm=llm, verbose=True)

agent.chat("What is 2 times 3?")

Relevant Logs/Tracbacks

---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
d:\Git\test_llama_index\notebook.ipynb Cell 18 line 1
----> 1 agent.chat("What is 2 times 3?")

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\callbacks\utils.py:39, in trace_method.<locals>.decorator.<locals>.wrapper(self, *args, **kwargs)
     37 callback_manager = cast(CallbackManager, callback_manager)
     38 with callback_manager.as_trace(trace_id):
---> 39     return func(self, *args, **kwargs)

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\agent\openai_agent.py:408, in BaseOpenAIAgent.chat(self, message, chat_history, tool_choice)
    397 @trace_method("chat")
    398 def chat(
    399     self,
   (...)
    402     tool_choice: Union[str, dict] = "auto",
    403 ) -> AgentChatResponse:
    404     with self.callback_manager.event(
    405         CBEventType.AGENT_STEP,
    406         payload={EventPayload.MESSAGES: [message]},
    407     ) as e:
--> 408         chat_response = self._chat(
    409             message, chat_history, tool_choice, mode=ChatResponseMode.WAIT
    410         )
    411         assert isinstance(chat_response, AgentChatResponse)
    412         e.on_end(payload={EventPayload.RESPONSE: chat_response})

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\agent\openai_agent.py:330, in BaseOpenAIAgent._chat(self, message, chat_history, tool_choice, mode)
    326     print(f"STARTING TURN {ix}\n---------------\n")
    327 llm_chat_kwargs = self._get_llm_chat_kwargs(
    328     openai_tools, current_tool_choice
    329 )
--> 330 agent_chat_response = self._get_agent_response(mode=mode, **llm_chat_kwargs)
    331 if not self._should_continue(self.latest_tool_calls, n_function_calls):
    332     logger.debug("Break: should continue False")

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\agent\openai_agent.py:292, in BaseOpenAIAgent._get_agent_response(self, mode, **llm_chat_kwargs)
    288 def _get_agent_response(
    289     self, mode: ChatResponseMode, **llm_chat_kwargs: Any
    290 ) -> AGENT_CHAT_RESPONSE_TYPE:
    291     if mode == ChatResponseMode.WAIT:
--> 292         chat_response: ChatResponse = self._llm.chat(**llm_chat_kwargs)
    293         return self._process_message(chat_response)
    294     elif mode == ChatResponseMode.STREAM:

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\llms\base.py:187, in llm_chat_callback.<locals>.wrap.<locals>.wrapped_llm_chat(_self, messages, **kwargs)
    178 with wrapper_logic(_self) as callback_manager:
    179     event_id = callback_manager.on_event_start(
    180         CBEventType.LLM,
    181         payload={
   (...)
    185         },
    186     )
--> 187     f_return_val = f(_self, messages, **kwargs)
    189     if isinstance(f_return_val, Generator):
    190         # intercept the generator and add a callback to the end
    191         def wrapped_gen() -> ChatResponseGen:

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\llms\openai.py:185, in OpenAI.chat(self, messages, **kwargs)
    183 else:
    184     chat_fn = completion_to_chat_decorator(self._complete)
--> 185 return chat_fn(messages, **kwargs)

File d:\Git\test_llama_index\.venv\Lib\site-packages\llama_index\llms\openai.py:238, in OpenAI._chat(self, messages, **kwargs)
    236 def _chat(self, messages: Sequence[ChatMessage], **kwargs: Any) -> ChatResponse:
    237     message_dicts = to_openai_message_dicts(messages)
--> 238     response = self._client.chat.completions.create(
    239         messages=message_dicts,
    240         stream=False,
    241         **self._get_model_kwargs(**kwargs),
    242     )
    243     openai_message = response.choices[0].message
    244     message = from_openai_message(openai_message)

File d:\Git\test_llama_index\.venv\Lib\site-packages\openai\_utils\_utils.py:299, in required_args.<locals>.inner.<locals>.wrapper(*args, **kwargs)
    297             msg = f"Missing required argument: {quote(missing[0])}"
    298     raise TypeError(msg)
--> 299 return func(*args, **kwargs)

File d:\Git\test_llama_index\.venv\Lib\site-packages\openai\resources\chat\completions.py:598, in Completions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_p, user, extra_headers, extra_query, extra_body, timeout)
    551 @required_args(["messages", "model"], ["messages", "model", "stream"])
    552 def create(
    553     self,
   (...)
    596     timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
    597 ) -> ChatCompletion | Stream[ChatCompletionChunk]:
--> 598     return self._post(
    599         "/chat/completions",
    600         body=maybe_transform(
    601             {
    602                 "messages": messages,
    603                 "model": model,
    604                 "frequency_penalty": frequency_penalty,
    605                 "function_call": function_call,
    606                 "functions": functions,
    607                 "logit_bias": logit_bias,
    608                 "max_tokens": max_tokens,
    609                 "n": n,
    610                 "presence_penalty": presence_penalty,
    611                 "response_format": response_format,
    612                 "seed": seed,
    613                 "stop": stop,
    614                 "stream": stream,
    615                 "temperature": temperature,
    616                 "tool_choice": tool_choice,
    617                 "tools": tools,
    618                 "top_p": top_p,
    619                 "user": user,
    620             },
    621             completion_create_params.CompletionCreateParams,
    622         ),
    623         options=make_request_options(
    624             extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
    625         ),
    626         cast_to=ChatCompletion,
    627         stream=stream or False,
    628         stream_cls=Stream[ChatCompletionChunk],
    629     )

File d:\Git\test_llama_index\.venv\Lib\site-packages\openai\_base_client.py:1055, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
   1041 def post(
   1042     self,
   1043     path: str,
   (...)
   1050     stream_cls: type[_StreamT] | None = None,
   1051 ) -> ResponseT | _StreamT:
   1052     opts = FinalRequestOptions.construct(
   1053         method="post", url=path, json_data=body, files=to_httpx_files(files), **options
   1054     )
-> 1055     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File d:\Git\test_llama_index\.venv\Lib\site-packages\openai\_base_client.py:834, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
    825 def request(
    826     self,
    827     cast_to: Type[ResponseT],
   (...)
    832     stream_cls: type[_StreamT] | None = None,
    833 ) -> ResponseT | _StreamT:
--> 834     return self._request(
    835         cast_to=cast_to,
    836         options=options,
    837         stream=stream,
    838         stream_cls=stream_cls,
    839         remaining_retries=remaining_retries,
    840     )

File d:\Git\test_llama_index\.venv\Lib\site-packages\openai\_base_client.py:877, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
    874     # If the response is streamed then we need to explicitly read the response
    875     # to completion before attempting to access the response text.
    876     err.response.read()
--> 877     raise self._make_status_error_from_response(err.response) from None
    878 except httpx.TimeoutException as err:
    879     if retries > 0:

BadRequestError: Error code: 400 - {'error': {'message': 'Unrecognized request arguments supplied: tool_choice, tools', 'type': 'invalid_request_error', 'param': None, 'code': None}}