k-NN: [BUG] OpenSearch 2.17 K-NN efficient filtering with a Date Range Filter No Results
What is the bug? We have a cluster with about 60 million documents spread across weekly indexes. The weekly indexes are all added to a “documents” alias. Each weekly index has about 1 million documents, and each doc has a faiss hnsw knn_vector field with 256 dimensions.
We upgraded to OpenSearch 2.17 last week, and some of our k-NN queries no longer return hits. We have isolated the issue to k-NN queries (that are using efficient filtering) that have a range filter for a date field.
Here is the vector field (embeddings.OAI_TE3L_256) mapping:
"embeddings": {
"properties": {
"OAI_TE3L_256": {
"type": "knn_vector",
"dimension": 256,
"method": {
"engine": "faiss",
"space_type": "l2",
"name": "hnsw",
"parameters": {
"m": 32
}
}
}
Here is an example query that returns 0 hits:
POST documents/_search
{
"from": 0,
"query": {
"knn": {
"embeddings.OAI_TE3L_256": {
"vector": [
-0.05308306,
-0.040473435,
-0.020121736,
-0.024970116,
0.017065149,
0.0007180346,
-0.041048344,
0.025315061,
-0.0696787,
-0.04986358,
0.11812417,
0.004086629,
-0.121267,
-0.03683236,
-0.063508034,
-0.10087697,
-0.1159012,
-0.032597214,
0.020332536,
-0.103329904,
0.15353844,
-0.05373462,
-0.000034284803,
-0.03058504,
0.04346295,
0.038748715,
-0.088458985,
0.03915115,
0.066957474,
0.14510646,
0.029665189,
-0.0040171607,
-0.079260476,
-0.022766307,
-0.06937208,
0.0005449637,
0.1864231,
-0.054577816,
0.010750757,
0.038729552,
0.06327807,
0.048292167,
0.015273356,
0.0040890244,
-0.07489119,
-0.009126645,
0.033459574,
-0.011258591,
-0.023954447,
0.009629688,
-0.03322961,
-0.03882537,
-0.070905164,
0.026349893,
0.121267,
-0.096431024,
0.0071336348,
0.13782431,
-0.034226116,
-0.10509296,
0.021673985,
-0.038729552,
-0.016672296,
0.061859965,
-0.06864387,
0.12824254,
0.030086787,
-0.05423287,
0.004139329,
-0.046644103,
0.0077324957,
0.08370642,
-0.091985084,
0.084856234,
0.017975418,
0.0033632044,
0.011910152,
-0.050630122,
0.116131164,
-0.08439631,
-0.030201769,
-0.06734075,
0.104862995,
0.04043511,
0.056379188,
0.0061515025,
0.064006284,
-0.23870128,
0.03117911,
-0.0060892208,
0.03867206,
-0.04162325,
0.00463279,
-0.11774091,
0.0556893,
-0.07496784,
-0.0155608095,
-0.0087960735,
-0.0011372374,
-0.0022026114,
-0.03468604,
-0.0071384255,
-0.010175849,
-0.08907222,
0.08255661,
-0.064159594,
-0.06829892,
-0.056417517,
-0.043386295,
-0.023494523,
-0.022517182,
0.00840322,
-0.038767878,
0.054616142,
-0.08439631,
0.06262651,
-0.027633851,
-0.03644909,
0.039017007,
-0.05047681,
0.026714,
-0.045954213,
0.10011043,
-0.03150489,
-0.026503202,
0.1031766,
-0.05756733,
0.029933479,
0.060135245,
-0.17323858,
-0.04150827,
0.043194663,
-0.09367147,
-0.033574555,
0.0029607697,
0.032482233,
0.034283604,
0.010607029,
-0.07044524,
-0.021942275,
-0.080486946,
0.036219127,
0.021348204,
-0.005686786,
-0.074086316,
-0.041776557,
-0.033267938,
-0.056915767,
-0.07719082,
0.07742078,
-0.1773779,
-0.058295544,
-0.052431498,
0.030182606,
-0.013251601,
-0.0072294525,
0.0696787,
-0.042849716,
0.019163558,
0.061131753,
0.074507914,
0.095511176,
-0.039285295,
0.100187086,
0.03825046,
-0.02579415,
0.13245852,
0.0035692127,
-0.022919616,
0.11375488,
-0.031485725,
-0.08133014,
-0.051818263,
-0.07059855,
-0.00053478306,
0.00737797,
0.056340862,
-0.11697436,
0.071556725,
0.0053514238,
0.1167444,
0.042888045,
0.025123425,
-0.0140852155,
-0.020351699,
0.0119963875,
-0.0105207935,
-0.031428237,
0.012695857,
-0.009773415,
-0.031370744,
-0.013558217,
-0.03526095,
-0.03656407,
-0.0095913615,
0.02100326,
0.053351346,
0.05825722,
-0.006208993,
-0.03665989,
0.07941379,
-0.09213839,
-0.067762345,
-0.058218893,
0.07174836,
0.09443802,
0.09006873,
-0.006343138,
-0.0071911253,
-0.12555964,
0.025698332,
0.014554723,
0.07339643,
0.0067072455,
0.02454852,
0.030144278,
0.01432476,
-0.025947459,
0.0651561,
0.039074495,
0.021463186,
-0.040090166,
-0.025104262,
-0.04346295,
-0.08753914,
-0.028400393,
0.09505125,
0.14012393,
-0.055804282,
0.013059965,
-0.022306383,
-0.0034662087,
-0.0038973887,
-0.120807074,
-0.06281815,
-0.047525626,
0.009073945,
0.04411451,
-0.062281564,
-0.055114396,
-0.0583722,
0.04032013,
0.00021514088,
-0.018780287,
-0.019048577,
-0.03284634,
-0.06856721,
0.069985315,
0.056839116,
-0.02127155,
0.06140004,
0.028745338,
-0.0561109,
0.033363756,
-0.025621679,
0.06872052
],
"k": 1000,
"filter": {
"bool": {
"filter": [
{
"bool": {
"must": [
{
"term": {
"documentType.keyword": "RAW"
}
},
{
"term": {
"contentLanguage.keyword": "English"
}
},
{
"exists": {
"field": "attachment"
}
},
{
"match_phrase_prefix": {
"topic.country": "Australia"
}
},
{
"range": {
"dateReceived": {
"gte": "2024-12-12T12:30:47.000Z"
}
}
}
]
}
}
]
}
}
}
}
}
}
The filtered subset for this query should result in 968 docs, which is less than k, and I believe should result in an exact knn search through the efficient knn filtering algorithm, however it does not seem like we are seeing this behavior. Instead, we are just getting 0 hits.
Overall, we are seeing inconsistent query results. For example:
- If we run this query on a single weekly index rather than the entire documents alias, we do get expected hits back.
- If we remove the range portion of the knn query filter, we get expected hits back
- If we move the range filter outside of the knn query filter (and transform the query into a knn post-filter instead), we do get hits back
- If we use a knn filter range query for a non-date field (such as a float field), we get expected hits back
What is the expected behavior? K-NN efficient filtering should work as described https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/ for knn range filters on a date field.
What is your host/environment?
- Opensearch 2.17
- k-NN plugin
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (6 by maintainers)
I see there is a PR in progress, thanks for continuing to work on solving this issue! I was wondering, is there any workaround for this issue? Like is there any way I could modify the query I used in this example https://github.com/opensearch-project/k-NN/issues/2339#issuecomment-2556096822 to get the correct results? Or if I modify the dates I’m indexing (like if I index docs with epochmilli date values instead), would that work around this issue?
I’m guessing the answer is there isn’t a workaround but just thought I’d ask 😃, otherwise I need to start the process of downgrading/migrating our cluster to an older version of Opensearch and wait until 2.19 is available in AWS.
Hi @kristib. Thanks for the data! With your commands I was able to reproduce the issue, will look into it! Additionally, I seem to get inconsistent results even when querying the individual indexes, so maybe not an alias problem.
Thanks @kristib for providing the reproducing steps. We will start looking into this. This could be an issue from Opensearch too, but we can say more once we complete the deep-dive(@buddharajusahil is doing the deep-dive).