spring-ai: Chat history breaks function calling

Bug description When chat history is enabled, the AI provides incorrect information about available functions, seemingly relying on outdated data from the chat history instead of querying the current available functions.

Environment

  • Java version: 21
  • Spring Boot version: 3.3
  • Vaadin version: 24
  • ChatClient implementation for AI interactions

Steps to reproduce

  1. Enable chat history in the application.
  2. Ask the AI, “What functions are available?”
  3. Modify the available functions.
  4. Ask the AI, “What functions are available?”
  5. Observe that the response includes incorrect or outdated functions that are not part of the currently available functions.
  6. Disable chat history.
  7. Ask the same question, “What functions are available?”
  8. Observe that the response correctly lists the current available functions.

Expected behavior When chat history is enabled, the AI should provide the correct list of available functions by querying the current data, rather than relying on outdated or incorrect information from the chat history.

Minimal Complete Reproducible example Below is a snippet of the code demonstrating the setup of the SpringAiAdapter and the issue. Notice that chat history is disabled.

public class SpringAiAdapter implements AiGateway {
    private final ChatClient chatClient;
    private final ProjectRepository projectRepository;
    private static final Logger log = LoggerFactory.getLogger(SpringAiAdapter.class);

    public SpringAiAdapter(ChatClient.Builder builder, ProjectRepository projectRepository, VectorStore vectorStore) {
        this.projectRepository = projectRepository;
        this.chatClient = builder.defaultSystem("""
                        You are an assistant project manager expert at managing many resources and schedules.
                        Adopt the user's tone to make them feel comfortable with you. If they are playful and silly, so are you. If they are professional and matter-of-fact, so are you.
                        Keep your responses short and direct because people need your help in a hurry, but for complex tasks, think out loud by writing each step.
                        For questions about long documents, pull the most relevant quote from the document and consider whether it answers the user's question or whether it lacks sufficient detail.
                        Today is {current_date}. This message was sent by {user_name} at exactly {message_creation_time}.
                        Available projects are: {available_projects}. The project name is its natural identifier.
                        When calling functions, always use the exact name of the project as provided here. For example, a user's request may reference `projct a`, `12345`, or simply `A`, but if `Project A (12345)` is on the list of available projects, then function calls should be made with `Project A (12345)`. However, if the user's request references a significantly different project name like `projct b`, `54333`, or simply `B`, then the request should be rejected.""")
                .defaultAdvisors(
                        // new VectorStoreChatMemoryAdvisor(vectorStore),
                        new LoggingAdvisor())
                .build();
    }

    @Override
    public Flux<String> sendMessageAndReceiveReplies(UserMessageDto userMessageDto) {
        String projectNames = String.join(", ", projectRepository.findAllNames());
        return chatClient.prompt()
                .system(sp -> sp.params(Map.of(
                        "current_date", LocalDate.now().toString(),
                        "message_creation_time", userMessageDto.creationTime().toString(),
                        "user_name", userMessageDto.userName(),
                        "available_projects", projectNames
                )))
                .functions("clockInFunction", "clockOutFunction", "findAllProjectNamesFunction", "updateProjectFunction")
                .user(userMessageDto.userMessageText())
                .advisors(advisorSpec -> advisorSpec.param(CHAT_MEMORY_CONVERSATION_ID_KEY, userMessageDto.chatId())
                        .param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))
                .stream()
                .content()
                .onErrorResume(WebClientResponseException.class, e -> {
                    log.error("WebClient request failed with status: {} and response body: {}",
                            e.getStatusCode(), e.getResponseBodyAsString(), e);
                    return Mono.error(new RuntimeException("Failed to communicate with AI service", e));
                });
    }
}

Reproduction output examples

With chat history disabled:

Nathan
Aug 31, 2024, 4:24 PM
What functions are available?
DroidComm
Aug 31, 2024, 4:24 PM
Here are the available functions:

1. **clockOutFunction** - Update an existing timesheet entry for a resource by adding a clock-out time.
2. **updateProjectFunction** - Update an existing timesheet entry's project.
3. **clockInFunction** - Create a new timesheet entry for a resource, optionally including a project name.
4. **findAllProjectNamesFunction** - Fetch the list of available project names.

Let me know how I can assist you with any of these!

With chat history enabled:

Nathan
Aug 31, 2024, 4:25 PM
What functions are available?
DroidComm
Aug 31, 2024, 4:25 PM
The available functions are:

1. **clockInFunction**: Create a new timesheet entry for a resource.
2. **updateProjectFunction**: Update an existing timesheet entry's project.
3. **findAllProjectNamesFunction**: Fetch the list of available project names.
4. **unfulfilledRequestFunction**: Handle requests or commands that the AI cannot fulfill by logging them as potential feature requests or bug reports.

If you need assistance with any of these functions, just let me know!

The response with chat history enabled shows an incorrect list of functions, including “unfulfilledRequestFunction,” which is not actually available in the current setup.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 18 (8 by maintainers)

Most upvoted comments

Disregard my previous message if you are using the latest build from source. Advisors and Functions look to be working as expected when building from source.

@LiveNathan since the tool and adviser APIs have changed significantly and I’m not sure if this still an issue or not. Here I’ve tried to refactor your example with the latest changes, though I might have missed something and most likely it wont work OOTB. Anyway if you are still facing related issue, perhaps it will help to reproduce it again.

/*
* Copyright 2025 - 2025 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.springframework.ai.chroma;

import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.function.Function;

import io.micrometer.observation.ObservationRegistry;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.chromadb.ChromaDBContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor;
import org.springframework.ai.chat.client.advisor.vectorstore.VectorStoreChatMemoryAdvisor;
import org.springframework.ai.chat.memory.ChatMemory;
import org.springframework.ai.chat.memory.MessageWindowChatMemory;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chroma.vectorstore.ChromaApi;
import org.springframework.ai.chroma.vectorstore.ChromaVectorStore;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.ai.openai.OpenAiEmbeddingModel;
import org.springframework.ai.openai.api.OpenAiApi;
import org.springframework.ai.retry.RetryUtils;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.ToolCallbackProvider;
import org.springframework.ai.tool.execution.DefaultToolExecutionExceptionProcessor;
import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor;
import org.springframework.ai.tool.resolution.DelegatingToolCallbackResolver;
import org.springframework.ai.tool.resolution.SpringBeanToolCallbackResolver;
import org.springframework.ai.tool.resolution.StaticToolCallbackResolver;
import org.springframework.ai.tool.resolution.ToolCallbackResolver;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Description;
import org.springframework.context.support.GenericApplicationContext;
import org.springframework.util.StringUtils;

import static org.assertj.core.api.Assertions.assertThat;
import static org.springframework.ai.chat.client.advisor.AbstractChatMemoryAdvisor.CHAT_MEMORY_CONVERSATION_ID_KEY;
import static org.springframework.ai.chat.client.advisor.AbstractChatMemoryAdvisor.CHAT_MEMORY_RETRIEVE_SIZE_KEY;

@SpringBootTest(classes = SpringAiIssue1301IT.Conf.class)
@EnabledIfEnvironmentVariable(named = "OPENAI_API_KEY", matches = ".+")
@Testcontainers
public class SpringAiIssue1301IT {

	private static final Logger log = LoggerFactory.getLogger(SpringAiIssue1301IT.class);

	@Container
	static ChromaDBContainer chromadb = new ChromaDBContainer(ChromaImage.DEFAULT_IMAGE);

	@Autowired
	protected ChatModel chatModel;

	@Autowired
	protected EmbeddingModel embeddingModel;

	@Autowired
	protected VectorStore vectorStore;

	@Test
	void functionCallTest() throws Exception {
		ChatMemory chatMemory = MessageWindowChatMemory.builder().build();

		var chatClient = ChatClient.builder(chatModel)
			.defaultSystem(
					"""
							You are an assistant project manager expert at managing many resources and schedules.
							Adopt the user's tone to make them feel comfortable register you. If they are playful and silly, so are you. If they are professional and matter-of-fact, so are you.
							Keep your responses short and direct because people need your help in a hurry, but for complex tasks, think out loud by writing each step.
							For questions about long documents, pull the most relevant quote from the document and consider whether it answers the user's question or whether it lacks sufficient detail.
							Today is {current_date}. This message was sent by {user_name} at exactly {message_creation_time} instant register {message_creation_timezone} timezone.
							Available projects are: {available_projects}. The project name is its natural identifier.""")
			.defaultToolNames("clockIn")
			.defaultAdvisors(new MessageChatMemoryAdvisor(chatMemory),
					VectorStoreChatMemoryAdvisor.builder(vectorStore).build())
			.build();

		String response = chatClient.prompt()
			.system(sp -> sp.params(Map.of("current_date", LocalDate.now().toString(), "message_creation_time",
					LocalDateTime.now(), "message_creation_timezone", ZoneId.systemDefault().toString(), "user_name",
					"Alice", "available_projects", List.of("Project A (12345)", "Project B (54333)"))))
			.user(u -> u.text("What functions are available? List the function names."))
			.advisors(advisorSpec -> advisorSpec.param(CHAT_MEMORY_CONVERSATION_ID_KEY, "696")
				.param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))
			.toolNames("clockIn")
			.call()
			.content();

		log.info("Response: {}", response);

		assertThat(response).contains("clockIn");

		// Update the available functions.
		response = chatClient.prompt()
			.system(sp -> sp.params(Map.of("current_date", LocalDate.now().toString(), "message_creation_time",
					LocalDateTime.now(), "message_creation_timezone", ZoneId.systemDefault().toString(), "user_name",
					"Alice", "available_projects", List.of("Project A (12345)", "Project B (54333)"))))
			.user(u -> u.text("What functions are available? List the function names."))
			.advisors(advisorSpec -> advisorSpec.param(CHAT_MEMORY_CONVERSATION_ID_KEY, "696")
				.param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))
			.toolNames("clockIn", "clockOut")
			.call()
			.content();

		// Assert that the response contains the new functions. Expect assertion to fail
		// as long as vector store is active.
		assertThat(response).contains("clockIn");
		assertThat(response).contains("clockOut");
	}

	@Configuration
	@EnableAutoConfiguration(exclude = { DataSourceAutoConfiguration.class })
	public static class Conf {

		record Request(String text) {
		}

		record Response(String text) {
		}

		@Bean
		@Description("Clock in")
		public Function<Request, Response> clockIn() {
			return request -> new Response("clockIn " + request.text());
		}

		@Bean
		@Description("Clock out")
		public Function<Request, Response> clockOut() {
			return request -> new Response("clockOut " + request.text());
		}

		@Bean
		public OpenAiApi openAiApi() {
			String apiKey = System.getenv("OPENAI_API_KEY");
			if (!StringUtils.hasText(apiKey)) {
				throw new IllegalArgumentException(
						"You must provide an API key.  Put it in an environment variable under the name OPENAI_API_KEY");
			}

			return OpenAiApi.builder().apiKey(apiKey).build();
		}

		@Bean
		public OpenAiChatModel openAiChatModel(OpenAiApi api, ToolCallingManager toolCallingManager) {
			return OpenAiChatModel.builder()
				.openAiApi(api)
				.defaultOptions(OpenAiChatOptions.builder().model(OpenAiApi.ChatModel.GPT_4_O_MINI).build())
				.retryTemplate(RetryUtils.DEFAULT_RETRY_TEMPLATE)
				.toolCallingManager(toolCallingManager)
				.build();
		}

		@Bean
		@ConditionalOnMissingBean
		ToolCallingManager toolCallingManager(ToolCallbackResolver toolCallbackResolver,
				ToolExecutionExceptionProcessor toolExecutionExceptionProcessor,
				ObjectProvider<ObservationRegistry> observationRegistry) {
			return ToolCallingManager.builder()
				.observationRegistry(observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP))
				.toolCallbackResolver(toolCallbackResolver)
				.toolExecutionExceptionProcessor(toolExecutionExceptionProcessor)
				.build();
		}

		@Bean
		@ConditionalOnMissingBean
		ToolCallbackResolver toolCallbackResolver(GenericApplicationContext applicationContext,
				List<ToolCallback> toolCallbacks, List<ToolCallbackProvider> tcbProviders) {

			List<ToolCallback> allFunctionAndToolCallbacks = new ArrayList<>(toolCallbacks);
			tcbProviders.stream()
				.map(pr -> List.of(pr.getToolCallbacks()))
				.forEach(allFunctionAndToolCallbacks::addAll);

			var staticToolCallbackResolver = new StaticToolCallbackResolver(allFunctionAndToolCallbacks);

			var springBeanToolCallbackResolver = SpringBeanToolCallbackResolver.builder()
				.applicationContext(applicationContext)
				.build();

			return new DelegatingToolCallbackResolver(
					List.of(staticToolCallbackResolver, springBeanToolCallbackResolver));
		}

		@Bean
		@ConditionalOnMissingBean
		ToolExecutionExceptionProcessor toolExecutionExceptionProcessor() {
			return new DefaultToolExecutionExceptionProcessor(false);
		}

		@Bean
		public OpenAiEmbeddingModel openAiEmbeddingModel(OpenAiApi api) {
			return new OpenAiEmbeddingModel(api);
		}

		@Bean
		@ConditionalOnMissingBean
		public SpringBeanToolCallbackResolver springAiFunctionManager(ApplicationContext context) {
			return SpringBeanToolCallbackResolver.builder()
				.applicationContext((GenericApplicationContext) context)
				.build();
		}

		@Bean
		public VectorStore vectorStore(EmbeddingModel embeddingModel, ChromaApi chromaApi) {
			return ChromaVectorStore.builder(chromaApi, embeddingModel).initializeSchema(true).build();
		}

		@Bean
		public ChromaApi chromaApi() {
			String chromaDbHost = chromadb.getHost();
			int chromaDbPort = chromadb.getMappedPort(8000); // Adjust port if necessary

			String baseUrl = String.format("http://%s:%d", chromaDbHost, chromaDbPort);
			return ChromaApi.builder().baseUrl(baseUrl).build();
		}

	}

}

I am seeing something similar when using the QuestionAnswerAdvisor or RetrievalAugmentationAdvisor and a custom function. The advisors run but the functions do not run. If I remove the advisors and re-issue the prompt, the functions are used.

        def checkAvailabilityFuncCallWrapper = FunctionCallbackWrapper.builder(new CheckAvailabilityTool())
    	.withName("CheckAvailability")
    	.withDescription("Returns true if rooms are available")
        .withResponseConverter((response) -> "" + response.available())
        .build()

        def advisor = RetrievalAugmentationAdvisor.builder().documentRetriever(this.vectorStoreDocumentRetriever).build()

        def response = this.chatClient.prompt().user(userPrompt).functions(checkAvailabilityFuncCallWrapper).advisors(advisor).call()
        def answer = response.content() 

I am using 1.0.0-SNAPSHOT

Hi @LiveNathan can you share please what Spring AI version are which chat model are us using?

Thanks for reporting this, we will look into it ASAP.