llama_index: [Question]: Index created from a LanceDBVectorStore is not returning all nodes?

Question Validation

  • I have searched both the documentation and discord for an answer.

Question

I don’t know what I am doing wrong. I chunked a few hundred documents using the HierarchicalNodeParser and stored them in a lanceDB database using this guide. It has about 24000 leaf nodes in it.

If I want to query the data, I use the code down below:

embed_model = OpenAIEmbedding(model="text-embedding-3-large")

db = lancedb.connect("./lancedb_RL_test_2")
Settings.embed_model = embed_model
table = db.open_table("docs_OpenAI")
vector_store = LanceDBVectorStore.from_table(table)
index = VectorStoreIndex.from_vector_store(vector_store)
storage_context = StorageContext.from_defaults(persist_dir="./RL_test_2")
base_retriever = index.as_retriever(similarity_top_k=10000)

question = "test"
nodes = base_retriever.retrieve(question)

What this seems to be doing is initially grabbing the same exact 1080 nodes from the database, then ranking them based on vector similarity to query. I tried tuning the overfetch_factor and nprobes parameters of the LanceDBVectorStore, but this seems to do nothing. I am very confused on what I might be doing wrong? Any help?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (7 by maintainers)

Most upvoted comments

that is not faulty logic, you are hardcoding the mode argument and we have added that as per user’s requirement/ input

@raghavdixit99 I think the problem @caes27 is pointing out is that "append" is not a valid mode for create_table (I see the below error when the table does not exist yet).

.venv/lib/python3.11/site-packages/lancedb/db.py", line 414, in create_table
    raise ValueError("mode must be either 'create' or 'overwrite'")

The llama index code is using the mode parameter for both create_table and table.add, but the values LanceDB expects for each are different. For create_table, valid modes are "create" or "overwrite", whereas for table.add, the mode must be "overwrite" or "append". This works OK for "overwrite" since the modes overlap, but it doesn’t work for "append".

Hey there, @caes27! I’m here to help you with any bugs, questions, or contributions you might have. Let’s tackle this together while we wait for a human maintainer.

The issue you’re encountering is likely due to the refine_factor parameter in the LanceDBVectorStore class. This parameter fetches more results than requested and then re-ranks them in memory. If this factor is set, it will fetch more nodes than the similarity_top_k value specified in your query.

Here’s the relevant code snippet:

table = self.connection.open_table(self.table_name)
lance_query = (
    table.search(
        query=query.query_embedding,
        vector_column_name=self.vector_column_name,
    )
    .limit(query.similarity_top_k)
    .where(where)
    .nprobes(self.nprobes)
)

if self.refine_factor is not None:
    lance_query.refine_factor(self.refine_factor)

In this code, if self.refine_factor is set, it will refine the results by reading extra elements and re-ranking them in memory. This could result in initially fetching 1080 nodes before any further ranking is applied [1][2].

To address this, you can try adjusting the refine_factor parameter when initializing the LanceDBVectorStore:

vector_store = LanceDBVectorStore.from_table(table, refine_factor=None)

Additionally, ensure that the nprobes parameter is set appropriately. The default value is 20, which balances accuracy and speed. You can adjust it based on your performance needs:

vector_store = LanceDBVectorStore.from_table(table, nprobes=20)

By fine-tuning these parameters, you should be able to control the number of nodes retrieved and improve the accuracy of your queries.

To continue talking to Dosu, mention @dosu.