llama_index: [Question]: Index created from a LanceDBVectorStore is not returning all nodes?
Question Validation
- I have searched both the documentation and discord for an answer.
Question
I don’t know what I am doing wrong. I chunked a few hundred documents using the HierarchicalNodeParser and stored them in a lanceDB database using this guide. It has about 24000 leaf nodes in it.
If I want to query the data, I use the code down below:
embed_model = OpenAIEmbedding(model="text-embedding-3-large")
db = lancedb.connect("./lancedb_RL_test_2")
Settings.embed_model = embed_model
table = db.open_table("docs_OpenAI")
vector_store = LanceDBVectorStore.from_table(table)
index = VectorStoreIndex.from_vector_store(vector_store)
storage_context = StorageContext.from_defaults(persist_dir="./RL_test_2")
base_retriever = index.as_retriever(similarity_top_k=10000)
question = "test"
nodes = base_retriever.retrieve(question)
What this seems to be doing is initially grabbing the same exact 1080 nodes from the database, then ranking them based on vector similarity to query. I tried tuning the overfetch_factor and nprobes parameters of the LanceDBVectorStore, but this seems to do nothing. I am very confused on what I might be doing wrong? Any help?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (7 by maintainers)
@raghavdixit99 I think the problem @caes27 is pointing out is that
"append"is not a valid mode forcreate_table(I see the below error when the table does not exist yet).The llama index code is using the
modeparameter for bothcreate_tableandtable.add, but the values LanceDB expects for each are different. For create_table, valid modes are"create"or"overwrite", whereas for table.add, the mode must be"overwrite"or"append". This works OK for"overwrite"since the modes overlap, but it doesn’t work for"append".Hey there, @caes27! I’m here to help you with any bugs, questions, or contributions you might have. Let’s tackle this together while we wait for a human maintainer.
The issue you’re encountering is likely due to the
refine_factorparameter in theLanceDBVectorStoreclass. This parameter fetches more results than requested and then re-ranks them in memory. If this factor is set, it will fetch more nodes than thesimilarity_top_kvalue specified in your query.Here’s the relevant code snippet:
In this code, if
self.refine_factoris set, it will refine the results by reading extra elements and re-ranking them in memory. This could result in initially fetching 1080 nodes before any further ranking is applied [1][2].To address this, you can try adjusting the
refine_factorparameter when initializing theLanceDBVectorStore:Additionally, ensure that the
nprobesparameter is set appropriately. The default value is20, which balances accuracy and speed. You can adjust it based on your performance needs:By fine-tuning these parameters, you should be able to control the number of nodes retrieved and improve the accuracy of your queries.
To continue talking to Dosu, mention @dosu.