usearch: Bug: index.search returns invalid keys when k > index size
Describe the bug
When creating a small index (size ~100) and searching it, I noticed it always returns some extremely large keys that are not actually present in the index. Here’s an example of my own print debugging output:
Warning: 92 samples not in index_labels: [94264669291728, 119, 9223372036854775808, 94264674115280, 94264674115312, 208, 94278552092195, 139415592574144, 9223372036854775808, 753, 94264669262496, 94264673534624, 94264668185904, 94264676614832, 94264661017696, 94264673538080, 94264676662528, 94264682064784, 94264681824960, 107, 94264655611472, 94264655611504, 94264676636832, 113, 94264676436544, 186, 94264675185232, 94264675185264, 94264662133040, 189, 94264676662400, 94264676662432, 94264676992352, 94264673533232, 94264672356160, 257, 94264674115376, 208, 94278552062388, 139415592574144, 6305, 94264669268912, 139415592574144, 94264676412336, 94264676412304, 208, 94278552063349, 139415592574144, 5345, 94264669268912, 139415592574144, 2256, 94264669235504, 94264669279184, 94264674131696, 94264680759264, 94264669315648, 94264669315680, 94264669259168, 94264669259200, 94264676396400, 94264676396432, 94264676396464, 94264676396496, 94264674051952, 94264674051984, 114, 9223372036854775808, 94264676462544, 208, 94278552066357, 139415592574144, 11457, 94264669235920, 139415592574144, 94261647247312, 94264674130320, 208, 94278552067317, 139415592574144, 10497, 94264669235920, 139415592574144, 94261647247309, 94264669280224, 182, 9223372036854775808, 23013835270, 94278552069446, 94264669295984, 94264669271152, 94278552069510]
On further debugging, I realized this happens when I search for more “neighbors” than there are points in the index.
Steps to reproduce
Here is how I get the error: I create an Index, embed some documents (around 100) and then insert them. When I search for > 100 neighbors, I get back invalid keys.
Expected behavior
index.search should only return keys that are present in the index. if searching for fewer items than there are in the index, i would expect to either only get back the items in the index (i.e. fewer than I asked for!), or else raise some kind of error message. silently returning invalid keys is not user-friendly! (in my opinion)
USearch version
2.11.3
Operating System
Debian slim-bookworm
Hardware architecture
x86
Which interface are you using?
Python bindings
Contact Details
andersonbcdefg@gmail.com
Is there an existing issue for this?
- I have searched the existing issues
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (7 by maintainers)
Commits related to this issue
- Test: Oversubscribed search Relates to #393 — committed to unum-cloud/usearch by ashvardanian 3 years ago
- Build: Released 2.11.4 [skip ci] ## [2.11.4](https://github.com/unum-cloud/usearch/compare/v2.11.3...v2.11.4) (2024-04-11) ### Fix * `#[repr(transparent)]` for `f16` and `b1x8` ([e182d77](https://g... — committed to unum-cloud/usearch by semantic-release-bot 3 years ago
I understand your concern – the case I shared is a toy example at local dev scale. My full scale app has multiple indices w/ millions of vectors. There are multiple indices because they are using fully different embedding spaces.