spring-ai: Chat history breaks function calling
Bug description When chat history is enabled, the AI provides incorrect information about available functions, seemingly relying on outdated data from the chat history instead of querying the current available functions.
Environment
- Java version: 21
- Spring Boot version: 3.3
- Vaadin version: 24
- ChatClient implementation for AI interactions
Steps to reproduce
- Enable chat history in the application.
- Ask the AI, “What functions are available?”
- Modify the available functions.
- Ask the AI, “What functions are available?”
- Observe that the response includes incorrect or outdated functions that are not part of the currently available functions.
- Disable chat history.
- Ask the same question, “What functions are available?”
- Observe that the response correctly lists the current available functions.
Expected behavior When chat history is enabled, the AI should provide the correct list of available functions by querying the current data, rather than relying on outdated or incorrect information from the chat history.
Minimal Complete Reproducible example
Below is a snippet of the code demonstrating the setup of the SpringAiAdapter and the issue. Notice that chat history is disabled.
public class SpringAiAdapter implements AiGateway {
private final ChatClient chatClient;
private final ProjectRepository projectRepository;
private static final Logger log = LoggerFactory.getLogger(SpringAiAdapter.class);
public SpringAiAdapter(ChatClient.Builder builder, ProjectRepository projectRepository, VectorStore vectorStore) {
this.projectRepository = projectRepository;
this.chatClient = builder.defaultSystem("""
You are an assistant project manager expert at managing many resources and schedules.
Adopt the user's tone to make them feel comfortable with you. If they are playful and silly, so are you. If they are professional and matter-of-fact, so are you.
Keep your responses short and direct because people need your help in a hurry, but for complex tasks, think out loud by writing each step.
For questions about long documents, pull the most relevant quote from the document and consider whether it answers the user's question or whether it lacks sufficient detail.
Today is {current_date}. This message was sent by {user_name} at exactly {message_creation_time}.
Available projects are: {available_projects}. The project name is its natural identifier.
When calling functions, always use the exact name of the project as provided here. For example, a user's request may reference `projct a`, `12345`, or simply `A`, but if `Project A (12345)` is on the list of available projects, then function calls should be made with `Project A (12345)`. However, if the user's request references a significantly different project name like `projct b`, `54333`, or simply `B`, then the request should be rejected.""")
.defaultAdvisors(
// new VectorStoreChatMemoryAdvisor(vectorStore),
new LoggingAdvisor())
.build();
}
@Override
public Flux<String> sendMessageAndReceiveReplies(UserMessageDto userMessageDto) {
String projectNames = String.join(", ", projectRepository.findAllNames());
return chatClient.prompt()
.system(sp -> sp.params(Map.of(
"current_date", LocalDate.now().toString(),
"message_creation_time", userMessageDto.creationTime().toString(),
"user_name", userMessageDto.userName(),
"available_projects", projectNames
)))
.functions("clockInFunction", "clockOutFunction", "findAllProjectNamesFunction", "updateProjectFunction")
.user(userMessageDto.userMessageText())
.advisors(advisorSpec -> advisorSpec.param(CHAT_MEMORY_CONVERSATION_ID_KEY, userMessageDto.chatId())
.param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))
.stream()
.content()
.onErrorResume(WebClientResponseException.class, e -> {
log.error("WebClient request failed with status: {} and response body: {}",
e.getStatusCode(), e.getResponseBodyAsString(), e);
return Mono.error(new RuntimeException("Failed to communicate with AI service", e));
});
}
}
Reproduction output examples
With chat history disabled:
Nathan
Aug 31, 2024, 4:24 PM
What functions are available?
DroidComm
Aug 31, 2024, 4:24 PM
Here are the available functions:
1. **clockOutFunction** - Update an existing timesheet entry for a resource by adding a clock-out time.
2. **updateProjectFunction** - Update an existing timesheet entry's project.
3. **clockInFunction** - Create a new timesheet entry for a resource, optionally including a project name.
4. **findAllProjectNamesFunction** - Fetch the list of available project names.
Let me know how I can assist you with any of these!
With chat history enabled:
Nathan
Aug 31, 2024, 4:25 PM
What functions are available?
DroidComm
Aug 31, 2024, 4:25 PM
The available functions are:
1. **clockInFunction**: Create a new timesheet entry for a resource.
2. **updateProjectFunction**: Update an existing timesheet entry's project.
3. **findAllProjectNamesFunction**: Fetch the list of available project names.
4. **unfulfilledRequestFunction**: Handle requests or commands that the AI cannot fulfill by logging them as potential feature requests or bug reports.
If you need assistance with any of these functions, just let me know!
The response with chat history enabled shows an incorrect list of functions, including “unfulfilledRequestFunction,” which is not actually available in the current setup.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 18 (8 by maintainers)
Disregard my previous message if you are using the latest build from source. Advisors and Functions look to be working as expected when building from source.
@LiveNathan since the tool and adviser APIs have changed significantly and I’m not sure if this still an issue or not. Here I’ve tried to refactor your example with the latest changes, though I might have missed something and most likely it wont work OOTB. Anyway if you are still facing related issue, perhaps it will help to reproduce it again.
I am seeing something similar when using the
QuestionAnswerAdvisororRetrievalAugmentationAdvisorand a custom function. The advisors run but the functions do not run. If I remove the advisors and re-issue the prompt, the functions are used.I am using
1.0.0-SNAPSHOTHi @LiveNathan can you share please what Spring AI version are which chat model are us using?
Thanks for reporting this, we will look into it ASAP.