llama_index: [Bug] `stream_chat()` does not trigger tool calling, though `chat()` does

Bug Description

I have a ReAct Agent (never tested it with an OpenAI Agent). I used to interact with it using .chat(). It was able to wield tools.

Today, I felt curious and replaced it with .stream_chat(). It stopped calling functions / using tools. It didn’t even seem to bother extract a thought-action-input triplet from the LLM generation.

Version

0.10.7; 0.9.43

Steps to Reproduce

I wrote a minimal reproducing script here. It uses identical settings, same query, and fixed temperature & seed for the LLM. It runs agent.stream_chat(QUERY) first and then agent.chat(QUERY), so you can compare the different behavior.

When you run:

PYTHONPATH=. python demo_for_issue.py

you’ll see:

>>>>>>>> With stream_chat:
[00:37:16] INFO     HTTP Request: POST http://localhost:11434/api/chat    _client.py:1013
                    "HTTP/1.1 200 OK"
>>>>>>>> Response:
>>>>>>>> With chat:
[00:37:18] INFO     HTTP Request: POST http://localhost:11434/api/chat    _client.py:1013
                    "HTTP/1.1 200 OK"
Thought: I need to use two tools to answer your question.
Action: about_the_user
Action Input: {'input': 'favorite drink'}

We can see that stream_chat didn’t trigger the “parse a tool use” procedure, while chat did.

Update: Also observed this with OpenAILike.

Relevant Logs/Tracbacks

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (3 by maintainers)

Most upvoted comments

I can also confirm that with a very simple RAG setup using Bedrock and a VectorStoreIndex that .chat runs the ReAct agent without a problem, but .stream_chat is not using the context at all. If I switch the mode to "context" in the index.as_chat_engine I do get answers based on the context for both .chat and .stream_chat.

Code to reproduce:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.bedrock import BedrockEmbedding
from llama_index.llms.bedrock import Bedrock

llm = Bedrock(
    temperature=0,
    model='anthropic.claude-3-sonnet-20240229-v1:0',
    region_name='us-east-1',
)
embed_model = BedrockEmbedding(
    model_id="amazon.titan-embed-text-v2:0",
    region_name="us-east-1"
)

reader = SimpleDirectoryReader(
    input_dir="./data_bob",
    recursive=True,
)

all_docs = []
for docs in reader.iter_data():
    for doc in docs:       
        all_docs.append(doc)

splitter = SentenceSplitter(chunk_size=1024)
index = VectorStoreIndex.from_documents(
    all_docs, transformations=[splitter], embed_model=embed_model
)

chat_engine = index.as_chat_engine(llm=llm)

question = "Who is bob?"

response = chat_engine.chat(question)
print("Chat response\n*******")
print(response)

stream_response = chat_engine.stream_chat(question)
print("\nStream chat response\n*******")
stream_response.print_response_stream()

The data_bob directory contains a single file whose contents is:

Bob is a civil engineer whose expertise is in creating interesting projects for other people.
He has three children and a cat.

The output of the code above:

Chat response
*******
Based on the information provided, Bob is a civil engineer who specializes in creating interesting projects. He has a family with three children and also owns a pet cat.

Stream chat response
*******
 I'm sorry, but I don't have enough context to determine exactly who "Bob" is referring to. Bob is a very common name, so without any additional details about the person, I cannot provide specifics about their identity, background, occupation, etc. If you could provide some more context about which Bob you are asking about, that would help me better understand and answer your question.

@logan-markewich I upgraded to the latest version of llama-index (0.10.43) but still not able to make it work. Did you run the code snippet that @omrihar provided without any modification, or did you change something?

I’m facing a similar issue on a simple RAG implementation, llm is Claude 3 Haiku with temperature=0:

image

Why the behaviour is so different?

hmm, very sus, will take a look