llama_index: Role 'tool' must be a response to a preceding message with 'tool_calls'
Bug Description
getting openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls' error when using json chat store with persistent paths, but when i checked the stored json tool_calls is saved before role tool and the chats are also saved by chat_store.
I am receiving this error in long chats but only when i am loading the chat store again for specific key, when i tested separately in while loop it works fine without error
Version
0.10.38
Steps to Reproduce
here is the api code
def stream_generator(generator, chat_store: SimpleChatStore):
yield from (json.dumps({"type": "content_block", "text": text}) for text in generator)
chat_store.persist(persist_path=CHAT_PERSIST_PATH)
@app.post("/chat")
async def chat(body: ChatRequest = Body()):
try:
if Path(CHAT_PERSIST_PATH).exists():
chat_store = SimpleChatStore.from_persist_path(CHAT_PERSIST_PATH)
else:
chat_store = SimpleChatStore()
memory = ChatMemoryBuffer.from_defaults(
chat_store=chat_store,
chat_store_key=body.chatId,
)
tool_spec = DataBaseToolSpec().to_tool_list()
agent = OpenAIAgent.from_tools(
tool_spec, llm=llm, verbose=True, system_prompt=system_prompt, memory=memory
)
response = agent.stream_chat(body.query)
return StreamingResponse(
stream_generator(response.response_gen, chat_store), media_type="application/x-ndjson"
)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e)) from e
i need to load the chat store on every request as want to save different chats and multiple user
Relevant Logs/Tracbacks
File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\llama_index\core\chat_engine\types.py", line 258, in response_gen
| raise self.exception
| File "C:\Users\anant\miniconda3\envs\super\Lib\threading.py", line 1073, in _bootstrap_inner
| self.run()
| File "C:\Users\anant\miniconda3\envs\super\Lib\threading.py", line 1010, in run
| self._target(*self._args, **self._kwargs)
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\llama_index\core\instrumentation\dispatcher.py", line 274, in wrapper
| result = func(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\llama_index\core\chat_engine\types.py", line 163, in write_response_to_history
| for chat in self.chat_stream:
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\llama_index\core\llms\callbacks.py", line 154, in wrapped_gen
| for x in f_return_val:
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\llama_index\llms\openai\base.py", line 454, in gen
| for response in client.chat.completions.create(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\openai\_utils\_utils.py", line 277, in wrapper
| return func(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\openai\resources\chat\completions.py", line 590, in create
| return self._post(
| ^^^^^^^^^^^
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\openai\_base_client.py", line 1240, in post
| return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\openai\_base_client.py", line 921, in request
| return self._request(
| ^^^^^^^^^^^^^^
| File "C:\Users\anant\miniconda3\envs\super\Lib\site-packages\openai\_base_client.py", line 1020, in _request
| raise self._make_status_error_from_response(err.response) from None
| openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.", 'type': 'invalid_request_error', 'param': 'messages.[1].role', 'code': None}}
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 23
Commits related to this issue
- fix: 对历史记录的toolcall验证是否成对, 参考: https://github.com/run-llama/llama_index/issues/13715 https://github.com/run-llama/llama_index/pull/16214 — committed to anka-afk/AstrBot-anka by anka-afk 2 years ago
Hey there, @anantgupta129! I’m here to help you out with any bugs, questions, or contributions you might have. Let’s work together to solve this issue.
The
openai.BadRequestErroryou’re encountering is due to the OpenAI API’s requirement that messages with the roletoolmust follow a preceding message with the roletool_calls. This error typically arises when the sequence of messages is not maintained correctly, especially when loading the chat store from a persistent path.Here are some steps to resolve this issue:
Ensure Correct Sequence of Messages: Verify that the sequence of messages in your chat store is correct. Messages with the role
toolshould always follow a message with the roletool_calls.Persist and Load Chat Store Correctly: When persisting and loading the chat store, ensure that the state is consistent and no messages are lost or reordered.
Use File Locking and Atomic Writes: Implement file locking and atomic writes to avoid race conditions and ensure data consistency when multiple requests are accessing the chat store concurrently.
Here is an updated version of your API code with these considerations:
Key Changes:
_lock) to ensure that only one thread can access the chat store at a time.By ensuring the correct sequence of messages and using file locking and atomic writes, you can avoid the
openai.BadRequestErrorand maintain a consistent state across multiple API requests [1].Details
About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
To resolve the
openai.BadRequestErrorthat states “Invalid parameter: messages with role ‘tool’ must be a response to a preceding message with ‘tool_calls’”, you need to ensure that any message with the role ‘tool’ is correctly preceded by a message with ‘tool_calls’. This can be managed by validating and structuring your chat messages properly.Here is a relevant function from the provided code that ensures only a single tool call is made, which might help in structuring your messages correctly:
Additionally, you should validate the response to ensure it meets the required structure:
Make sure to call
_validate_chat_with_tools_responseafter receiving the response to ensure it is correctly structured. This should help in resolving theBadRequestErrorby ensuring the message sequence is valid [1].To continue talking to Dosu, mention @dosu.
I have encountered the same with Autogen agents too, while resuming that chat.
This seems to be an error with OpenAI’s API that incorrectly flags valid sequences as erroneous.
this is the code i ran separately, on loading the chat store once this error is not coming