llama_index: [Bug] `stream_chat()` does not trigger tool calling, though `chat()` does
Bug Description
I have a ReAct Agent (never tested it with an OpenAI Agent). I used to interact with it using .chat(). It was able to wield tools.
Today, I felt curious and replaced it with .stream_chat(). It stopped calling functions / using tools. It didn’t even seem to bother extract a thought-action-input triplet from the LLM generation.
Version
0.10.7; 0.9.43
Steps to Reproduce
I wrote a minimal reproducing script here. It uses identical settings, same query, and fixed temperature & seed for the LLM. It runs agent.stream_chat(QUERY) first and then agent.chat(QUERY), so you can compare the different behavior.
When you run:
PYTHONPATH=. python demo_for_issue.py
you’ll see:
>>>>>>>> With stream_chat:
[00:37:16] INFO HTTP Request: POST http://localhost:11434/api/chat _client.py:1013
"HTTP/1.1 200 OK"
>>>>>>>> Response:
>>>>>>>> With chat:
[00:37:18] INFO HTTP Request: POST http://localhost:11434/api/chat _client.py:1013
"HTTP/1.1 200 OK"
Thought: I need to use two tools to answer your question.
Action: about_the_user
Action Input: {'input': 'favorite drink'}
We can see that stream_chat didn’t trigger the “parse a tool use” procedure, while chat did.
Update: Also observed this with OpenAILike.
Relevant Logs/Tracbacks
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (3 by maintainers)
I can also confirm that with a very simple RAG setup using
Bedrockand aVectorStoreIndexthat.chatruns the ReAct agent without a problem, but.stream_chatis not using the context at all. If I switch the mode to"context"in theindex.as_chat_engineI do get answers based on the context for both.chatand.stream_chat.Code to reproduce:
The
data_bobdirectory contains a single file whose contents is:The output of the code above:
@logan-markewich I upgraded to the latest version of llama-index (0.10.43) but still not able to make it work. Did you run the code snippet that @omrihar provided without any modification, or did you change something?
I’m facing a similar issue on a simple RAG implementation, llm is Claude 3 Haiku with
temperature=0:Why the behaviour is so different?
hmm, very sus, will take a look