k-NN: [BUG] Insufficient number of hits for nested knn queries with efficient filter
Hello π
What is the bug?
We use an index to store text documents for semantic search purpose. The text being long, we chunk it in paragraph to embed it using all-MiniLM-L6-v2 model. Each chunk being stored in that nested field of the document.
Each document has also an account_id attribute that we use when querying (efficient filtering).
Then we do approximative knn queries with lucene hnsw.
From these documentations :
- k-NN search with nested fields
- k-NN search with filters
- and mosty Enhanced multi-vector support for OpenSearch k-NN search with nested fields
I expect when executing a knn query on this nested field with efficient filter to get at least n hits, n being the minimum between k and the number of documents that match the efficient filter.
But for some specific input vector or query_text, we get less than n hits, and sometimes even 0. For the same filter with a different query, we get the correct n hits.
We have two other indices without nested field (only one vector per document) with the same efficient filter and it works as expected.
Seems similar to this https://github.com/opensearch-project/k-NN/issues/2222 or https://github.com/opensearch-project/k-NN/issues/2339 except the efficient filtering is as simple as a term filter.
How can one reproduce the bug?
Error happens on specific queries so itβs hard to reproduce.
Here is the mapping of the index :
{
"knowledge-index": {
"mappings": {
"properties": {
"accountId": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"metadata": {
"type": "text"
},
"metadataEmbedding": {
"type": "nested",
"properties": {
"knn": {
"type": "knn_vector",
"dimension": 384,
"method": {
"engine": "lucene",
"space_type": "l2",
"name": "hnsw",
"parameters": {}
}
}
}
},
"timestamp": {
"type": "date"
}
}
}
}
}
Here is the query :
GET /knowledge-index/_search?preference=_primary&explain=true&request_cache=false
{
"from": 0,
"_source": {
"excludes": [
"metadataEmbedding"
]
},
"query": {
"nested": {
"score_mode": "max",
"path": "metadataEmbedding",
"query": {
"neural": {
"metadataEmbedding.knn": {
"query_text": "<query_text>",
"model_id": "9QxR8YsBSCN1wquQEH2b",
"k": <k>,
"filter": {
"term": {
"accountId": "<account_id>"
}
}
}
}
}
}
}
}
For k = 38, I get 6 hits
"hits": {
"total": {
"value": 6,
"relation": "eq"
},
"max_score": 0.50342417,
But for k = 1000 I get 32 hits, and k = 10000 (max value) 232 hits.
For another query_text value, I have different results where hits is always = k (or the max of documents that match filter which is 232)
I have the same results when converting first the text in vector and use directly the vector without the neural instruction :
POST /_plugins/_ml/_predict/text_embedding/9QxR8YsBSCN1wquQEH2b
{
"text_docs":[ "<query_text>"],
"return_number": true,
"target_response": ["sentence_embedding"]
}
GET /knowledge-index/_search?preference=_primary&explain=true&request_cache=false
{
"size": 5,
"_source": {
"excludes": [
"metadataEmbedding"
]
},
"query": {
"nested": {
"path": "metadataEmbedding",
"query": {
"knn": {
"metadataEmbedding.knn": {
"vector": [
....
],
"k": 38,
"filter": {
"term": {
"accountId": "<account_id>"
}
}
}
}
}
}
}
}
What is the expected behavior?
Getting n hits, n being the minimum between k and the number of documents that match the efficient filter.
What is your host/environment?
- opensearch version : 2.17.1
Do you have any additional context?
Here is the result of GET /_plugins/_knn/stats?pretty on the node :
{
"max_distance_query_with_filter_requests": 0,
"graph_memory_usage_percentage": 0,
"graph_query_requests": 0,
"graph_memory_usage": 0,
"cache_capacity_reached": false,
"load_success_count": 0,
"training_memory_usage": 0,
"indices_in_cache": {},
"script_query_errors": 0,
"hit_count": 0,
"knn_query_requests": 2215,
"total_load_time": 0,
"miss_count": 0,
"min_score_query_requests": 0,
"knn_query_with_filter_requests": 2215,
"training_memory_usage_percentage": 0,
"max_distance_query_requests": 0,
"lucene_initialized": true,
"graph_index_requests": 0,
"faiss_initialized": false,
"load_exception_count": 0,
"training_errors": 0,
"min_score_query_with_filter_requests": 0,
"eviction_count": 0,
"nmslib_initialized": false,
"script_compilations": 1,
"script_query_requests": 2,
"graph_stats": {
"refresh": {
"total_time_in_millis": 0,
"total": 0
},
"merge": {
"current": 0,
"total": 0,
"total_time_in_millis": 0,
"current_docs": 0,
"total_docs": 0,
"total_size_in_bytes": 0,
"current_size_in_bytes": 0
}
},
"graph_query_errors": 0,
"indexing_from_model_degraded": false,
"graph_index_errors": 0,
"training_requests": 0,
"script_compilation_errors": 0
},
Any idea on what could be the issue here ? Am I right to expect k hits for nested fields with efficient filter ?
Thanks for your help.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (4 by maintainers)
Yes