spring-ai: ElasticSearch doSimilaritySearch broken after recent changes in Document
After update to the most recent version the similarity search gives an exception because the Document is no longer compatible to the data return by the search hits
- Document (org.springframework.ai.document.Document) is directly serialized from the ES response using standard jackson map to object serialization
doSimilaritySearch(SearchRequest searchRequest)
...
SearchResponse<Document> res = this.elasticsearchClient.search(
sr -> sr.index(this.options.getIndexName())
.knn(knn -> knn.queryVector(EmbeddingUtils.toList(vectors))
.similarity(finalThreshold)
.k((long) searchRequest.getTopK())
.field("embedding")
.numCandidates((long) (1.5 * searchRequest.getTopK()))
.filter(fl -> fl.queryString(
qs -> qs.query(getElasticsearchQueryString(searchRequest.getFilterExpression()))))),
----> Document.class);
return res.hits().hits().stream().map(this::toDocument).collect(Collectors.toList());
- The previous Document class was compatible with ES responses
"_source":{
"embedding":[…],
"content": "...",
"media":[],
"metadata":{ .... }
Now, it isn´t anymore in the Document class, after #1794. context has been renamed to text, media is no longer an array
- Therefore I get the error
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `org.springframework.ai.model.Media` from Array value (token `JsonToken.START_ARRAY`)
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 8428] (through reference chain: org.springframework.ai.document.Document["media"])
I’m not able to use ElasticSearch queries through the Vector Store.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (7 by maintainers)
@dev-jonghoonpark ,
I noticed that in your last commit you removed ElasticSearchDocument in order to manipulate the fields as a map to change the embedding field name. That will make the toDocument harder to implment. I did that at first : use Object as the parameterized type in the search, and then toDocument receives a map of values, but the the solution is not strongly typed and its based on strings 😦
So I think the ElasticSearchDocument should be reintroduced (at least only for reading), if the configurable name for embeddings field cannot be solved by any Spring technique (like a dynamic json annotation or mixin) or in the Jackson mapper.