llama_index: [Bug]: Azure Search Client does not get propagated via VectorStoreIndex
Bug Description
vector_store_address= <azure endpoint> vector_store_password= <azure api key> search_client = SearchClient( endpoint=vector_store_address, index_name=INDEX_NAME, credential=AzureKeyCredential(vector_store_password) )
Define a query that matches all documents
query = “*”
Set the API version
api_version = <api version>
search_mode = “all”
Search for documents with the query
response = search_client.search( search_text=query, search_mode=search_mode )
docs = [i for i in response]
llama_docs = []
for doc in docs:
llama_docs.append(
llama_doc(
id_=doc[‘id’],
text= doc[‘content’],
metadata={
“filename”: doc[‘filename’],
“file_path”: doc[‘file_path’],
“a”: doc[‘a’],
“b”: doc[‘b’],
“c”: doc[‘c’],
“page_number”: doc[‘page_number’],
},
)
)
Settings.embed_model = AzureOpenAIEmbedding( model=conf.openai_embedding_model, deployment_name=conf.openai_embedding_model, api_key=conf.openai_api_key, api_base=conf.openai_endpoint, azure_endpoint=conf.openai_endpoint, api_version=conf.openai_version, async_http_client=httpx.AsyncClient(proxies=<proxy>), http_client=httpx.Client(proxies=<proxy>), )
Settings.llm = AzureOpenAI( engine=OPENAI_GPT_4, model=OPENAI_GPT_4, temperature=0.0, azure_endpoint=conf.openai_endpoint, api_key=conf.openai_api_key, api_version=conf.openai_version, http_client=httpx.Client(proxies=<proxy>), )
sample_llama_docs = random.sample(llama_docs, 50)
llama_vector_store = AzureAISearchVectorStore( search_or_index_client=search_client, filterable_metadata_field_keys=[‘filename’, ‘a’, ‘b’, ‘c’], id_field_key=“id”, chunk_field_key=“content”, embedding_field_key=“content_vector”, doc_id_field_key=“id”, metadata_string_field_key=“metadata” )
storage_context = StorageContext.from_defaults(vector_store=llama_vector_store)
index = VectorStoreIndex.from_documents( [], storage_context=storage_context, embed_model=Settings.embed_model, )
relevant_ids = [i for i in dict(qa_dataset)[‘relevant_docs’].keys()]
g,s,f=qa_dataset.relevant_docs[relevant_ids[0]][0].split(‘_’)[:3]
a_filter = MetadataFilter(key=a, operator=FilterOperator.EQUAL_TO, value=g) b_filter = MetadataFilter(key=b, operator=FilterOperator.EQUAL_TO, value=s) c_filter = MetadataFilter(key=c, operator=FilterOperator.EQUAL_TO, value=f)
filter_names = [a_filter, b_filter, c_filter] filters = MetadataFilters(filters=filter_names, condition=FilterCondition.AND)
test_llama_retriver = index.as_retriever(similarity_top_k=3, filters=filters)
this works
print(test_llama_retriver.retrieve(‘Whats the name?’)) print(“\n\n”)
retriever_evaluator = RetrieverEvaluator.from_metric_names( [“mrr”, “hit_rate”, “precision”, “recall”], retriever=test_llama_retriver )
sample_id=relevant_ids[0] sample_query = qa_dataset.queries[sample_id] sample_expected = qa_dataset.relevant_docs[sample_id]
nest_asyncio.apply()
eval_result = retriever_evaluator.evaluate(sample_query,sample_expected)
print(eval_result)
Issue: eval_result = retriever_evaluator.evaluate(sample_query,sample_expected) triggers "AttributeError: ‘NoneType’ object has no attribute ‘search’
After tracing the code, the below function causes the issue -
`async def _acreate_query_result( self, search_query: str, vectors: Optional[List[Any]] ) -> VectorStoreQueryResult: results = await self._search_client.search( search_text=search_query, vector_queries=vectors, top=self._query.similarity_top_k, select=self._select_fields, filter=self._odata_filter, )
id_result = []
node_result = []
score_result = []
async for result in results:
node_id = result[self._field_mapping["id"]]
metadata_str = result[self._field_mapping["metadata"]]
metadata = json.loads(metadata_str) if metadata_str else {}
score = result["@search.score"]
chunk = result[self._field_mapping["chunk"]]
try:
node = metadata_dict_to_node(metadata)
node.set_content(chunk)
except Exception:
# NOTE: deprecated legacy logic for backward compatibility
metadata, node_info, relationships = legacy_metadata_dict_to_node(
metadata
)
node = TextNode(
text=chunk,
id_=node_id,
metadata=metadata,
start_char_idx=node_info.get("start", None),
end_char_idx=node_info.get("end", None),
relationships=relationships,
)
logger.debug(f"Retrieved node id {node_id} with node data of {node}")
id_result.append(node_id)
node_result.append(node)
score_result.append(score)
logger.debug(
f"Search query '{search_query}' returned {len(id_result)} results."
)
return VectorStoreQueryResult(
nodes=node_result, similarities=score_result, ids=id_result
)
`
This is because I need to convert AzureAISearchVectorStore to a VectorStoreIndex object to use as_retriever and the search_client passed to instantiate AzureAISearchVectorStore does not get propagated to VectorStoreIndex
I even tried manually overriding index._search_client = search_client but does not work
llama-index versions:
| Package | Version |
|---|---|
| llama-index | 0.10.62 |
| llama-index-agent-openai | 0.2.9 |
| llama-index-cli | 0.1.13 |
| llama-index-core | 0.10.62 |
| llama-index-embeddings-azure-openai | 0.1.11 |
| llama-index-embeddings-openai | 0.1.11 |
| llama-index-indices-managed-llama-cloud | 0.2.7 |
| llama-index-legacy | 0.9.48 |
| llama-index-llms-azure-openai | 0.1.10 |
| llama-index-llms-openai | 0.1.29 |
| llama-index-multi-modal-llms-openai | 0.1.9 |
| llama-index-program-openai | 0.1.7 |
| llama-index-question-gen-openai | 0.1.3 |
| llama-index-readers-file | 0.1.32 |
| llama-index-readers-llama-parse | 0.1.6 |
| llama-index-vector-stores-azureaisearch | 0.1.13 |
Version
0.10.62
Steps to Reproduce
vector_store_address= <azure endpoint> vector_store_password= <azure api key> search_client = SearchClient( endpoint=vector_store_address, index_name=INDEX_NAME, credential=AzureKeyCredential(vector_store_password) )
Define a query that matches all documents
query = “*”
Set the API version
api_version = <api version>
search_mode = “all”
Search for documents with the query
response = search_client.search( search_text=query, search_mode=search_mode )
docs = [i for i in response]
llama_docs = []
for doc in docs:
llama_docs.append(
llama_doc(
id_=doc[‘id’],
text= doc[‘content’],
metadata={
“filename”: doc[‘filename’],
“file_path”: doc[‘file_path’],
“a”: doc[‘a’],
“b”: doc[‘b’],
“c”: doc[‘c’],
“page_number”: doc[‘page_number’],
},
)
)
Settings.embed_model = AzureOpenAIEmbedding( model=conf.openai_embedding_model, deployment_name=conf.openai_embedding_model, api_key=conf.openai_api_key, api_base=conf.openai_endpoint, azure_endpoint=conf.openai_endpoint, api_version=conf.openai_version, async_http_client=httpx.AsyncClient(proxies=<proxy>), http_client=httpx.Client(proxies=<proxy>), )
Settings.llm = AzureOpenAI( engine=OPENAI_GPT_4, model=OPENAI_GPT_4, temperature=0.0, azure_endpoint=conf.openai_endpoint, api_key=conf.openai_api_key, api_version=conf.openai_version, http_client=httpx.Client(proxies=<proxy>), )
sample_llama_docs = random.sample(llama_docs, 50)
llama_vector_store = AzureAISearchVectorStore( search_or_index_client=search_client, filterable_metadata_field_keys=[‘filename’, ‘a’, ‘b’, ‘c’], id_field_key=“id”, chunk_field_key=“content”, embedding_field_key=“content_vector”, doc_id_field_key=“id”, metadata_string_field_key=“metadata” )
storage_context = StorageContext.from_defaults(vector_store=llama_vector_store)
index = VectorStoreIndex.from_documents( [], storage_context=storage_context, embed_model=Settings.embed_model, )
relevant_ids = [i for i in dict(qa_dataset)[‘relevant_docs’].keys()]
g,s,f=qa_dataset.relevant_docs[relevant_ids[0]][0].split(‘_’)[:3]
a_filter = MetadataFilter(key=a, operator=FilterOperator.EQUAL_TO, value=g) b_filter = MetadataFilter(key=b, operator=FilterOperator.EQUAL_TO, value=s) c_filter = MetadataFilter(key=c, operator=FilterOperator.EQUAL_TO, value=f)
filter the retriever to respond from relevant document chunks
filter_names = [a_filter, b_filter, c_filter] filters = MetadataFilters(filters=filter_names, condition=FilterCondition.AND)
test_llama_retriver = index.as_retriever(similarity_top_k=3, filters=filters)
this works
print(test_llama_retriver.retrieve(‘Whats the patients name?’)) print(“\n\n”)
retriever_evaluator = RetrieverEvaluator.from_metric_names( [“mrr”, “hit_rate”, “precision”, “recall”], retriever=test_llama_retriver )
sample_id=relevant_ids[0] sample_query = qa_dataset.queries[sample_id] sample_expected = qa_dataset.relevant_docs[sample_id]
nest_asyncio.apply()
eval_result = retriever_evaluator.evaluate(sample_query,sample_expected)
print(eval_result)
Relevant Logs/Tracbacks
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18
Hey there, @psureshmagadi17! I’m here to help you out. If you need assistance with bugs, questions, or contributions, feel free to ask.
To ensure that the
search_clientis correctly propagated to theVectorStoreIndexand avoid theAttributeError: 'NoneType' object has no attribute 'search'when usingRetrieverEvaluator, you need to make sure that thesearch_clientis properly initialized and assigned within theAzureAISearchVectorStoreclass.Here is the relevant part of the
AzureAISearchVectorStoreclass constructor that handles the initialization of thesearch_client:Additionally, you can refer to the test cases provided in the
llama-index-legacy/tests/vector_stores/test_azureaisearch.pyfile to ensure that thesearch_clientis correctly mocked and tested. Here is an example of how to create a mock vector store and test the initialization:Ensure that the
search_or_index_clientis correctly passed and that thesearch_clientis properly initialized within the constructor. This will prevent theNoneTypeerror when theRetrieverEvaluatortries to use thesearch_client. Additionally, make sure to handle both synchronous and asynchronous clients as shown in the updated constructor and test cases [1][2][3][4].To continue talking to Dosu, mention @dosu.
To ensure that the
search_clientis correctly maintained throughout the asynchronous calls, especially when usingRetrieverEvaluator, you need to explicitly pass thesearch_clientto the asynchronous context or ensure that theAzureQueryResultSearchBaseclass and its derived classes correctly maintain theself._search_clientthroughout the asynchronous calls.Here are the detailed steps and code modifications required:
Step 1: Modify the
AzureQueryResultSearchBaseClass to Include an Asynchronous Method_acreate_query_resultStep 2: Ensure Derived Classes Use the Asynchronous Method
Step 3: Add the
asearchMethod to the Derived ClassesBy following these steps, you ensure that the
search_clientis correctly maintained and used in both synchronous and asynchronous contexts, allowing theRetrieverEvaluatorto function properly.Additionally, you can refer to the
BaseRetrieverclass from the LlamaIndex framework for an example of how to handle both synchronous and asynchronous retrieval methods. TheBaseRetrieverclass uses_retrieve_from_objectand_aretrieve_from_objectto manage synchronous and asynchronous retrievals, respectively, and providesretrieveandaretrievemethods to handle retrieval in both contexts. This approach ensures that the client is correctly maintained and used throughout the retrieval process. Here is a relevant excerpt from theBaseRetrieverclass:This structure ensures that both synchronous and asynchronous retrievals are handled appropriately, maintaining the integrity of the
search_clientthroughout the process [1][2][3].To continue talking to Dosu, mention @dosu.
To ensure that the
search_clientis correctly propagated to theVectorStoreIndexand subsequently to the retriever, you need to make sure that theAzureAISearchVectorStoreis properly initialized and that thesearch_clientis accessible within theVectorStoreIndex.Here are the steps to ensure proper propagation:
Initialize
AzureAISearchVectorStorewithsearch_client:Create
StorageContextandVectorStoreIndex:Configure Filters and Create Retriever:
Pass Retriever to
RetrieverEvaluator:Key Points to Ensure Proper Propagation:
search_clientis correctly initialized and passed to theAzureAISearchVectorStore.AzureAISearchVectorStoreclass should have a property to access thesearch_client. This is typically done using a property method:By following these steps, you ensure that the
search_clientis correctly propagated and used within theAzureAISearchVectorStoreclass, and subsequently within theVectorStoreIndexand the retriever [1][2].To continue talking to Dosu, mention @dosu.
Based on the detailed trace and the provided code flow, it appears that the
self._search_clientis correctly set during the initialization and retrieval process but gets wiped out during the evaluation process withRetrieverEvaluator.Here are the key points and potential issues to investigate:
Initialization of
AzureAISearchVectorStore:self._search_clientis correctly set during the initialization ofAzureAISearchVectorStore.Retrieval Process:
self._search_clientis accessible and correctly used during the retrieval process withtest_llama_retriever.retrieve('Whats the name?').Evaluation Process:
RetrieverEvaluator.evaluate(sample_query, sample_expected).Potential Issue:
The
self._search_clientmight be getting reset or not properly passed during the asynchronous evaluation process. This could be due to the way the asynchronous methods are handling theself._search_client.Steps to Debug and Fix:
Ensure Proper Initialization in Asynchronous Methods:
self._search_clientis correctly passed and maintained in all asynchronous methods.Check Asynchronous Method Calls:
_acreate_query_resultmethod inAzureQueryResultSearchBaseto ensure thatself._search_clientis not getting reset or lost.Example Debugging Steps:
self._search_clientin Asynchronous Methods:_acreate_query_resultmethod to check the state ofself._search_client.self._search_clientis Passed Correctly:AzureQueryResultSearchBaseclass and its derived classes correctly maintain theself._search_clientthroughout the asynchronous calls.Example Fix:
If the issue is due to the asynchronous context losing the
self._search_client, you might need to ensure that theself._search_clientis explicitly passed or maintained in the asynchronous context.By following these steps, you can identify where the
self._search_clientis getting lost and ensure it remains set throughout the evaluation process. This will help maintain the correct state of theself._search_clientduring the evaluation withRetrieverEvaluator[1][2][3].To continue talking to Dosu, mention @dosu.
To ensure that
self._search_clientremains set throughout the evaluation process when usingRetrieverEvaluatorwithAzureAISearchVectorStoreandVectorStoreIndex, you need to make sure thatself._search_clientis properly initialized in theAzureAISearchVectorStoreclass. The initialization ofself._search_clientis handled in the__init__method of theAzureAISearchVectorStoreclass.Here is the relevant part of the
__init__method that ensuresself._search_clientis set:This code ensures that
self._search_clientis properly initialized based on the type ofsearch_or_index_clientprovided. Ifsearch_or_index_clientis of typeSearchIndexClient, it initializesself._search_clientusing theget_search_clientmethod. Ifsearch_or_index_clientis of typeSearchClient, it directly assigns it toself._search_client.By ensuring that
self._search_clientis correctly initialized in the constructor, it will remain set throughout the evaluation process [1][2].To continue talking to Dosu, mention @dosu.
To ensure that the
search_clientremains correctly set and does not get wiped out during the evaluation process when usingRetrieverEvaluatorwithAzureAISearchVectorStoreandVectorStoreIndex, you should ensure that thesearch_clientis properly initialized and maintained throughout the lifecycle of theAzureAISearchVectorStoreinstance.Here is the relevant part of the
AzureAISearchVectorStoreclass that initializes and maintains thesearch_client:Additionally, you can refer to the test cases provided in the
llama-index-legacy/tests/vector_stores/test_azureaisearch.pyfile to understand how to mock and test thesearch_clientinitialization and usage:Ensure that the
search_clientis properly initialized in the constructor and accessed via theclientproperty. This setup, along with the provided test cases, will help maintain thesearch_clientthroughout the lifecycle of theAzureAISearchVectorStoreinstance [1][2][3][4].To continue talking to Dosu, mention @dosu.
To ensure that the
search_clientis correctly propagated from theAzureAISearchVectorStoreto theVectorStoreIndexand subsequently to theVectorIndexRetriever, you need to verify that theVectorStoreIndexcorrectly utilizes thesearch_clientfrom theAzureAISearchVectorStore.Here are the steps to ensure proper propagation:
Ensure
AzureAISearchVectorStoreInitialization:Create
StorageContextandVectorStoreIndex:Configure Filters and Create Retriever:
Pass Retriever to
RetrieverEvaluator:Key Points to Ensure Proper Propagation:
search_clientis correctly initialized and passed to theAzureAISearchVectorStore.AzureAISearchVectorStoreclass should have a property to access thesearch_client. This is typically done using a property method:Verify Propagation in
VectorStoreIndex:Ensure that the
VectorStoreIndexcorrectly utilizes thesearch_clientfrom theAzureAISearchVectorStore. TheVectorStoreIndexshould have access to thesearch_clientthrough thestorage_context.Example:
By following these steps, you ensure that the
search_clientis correctly propagated and used within theAzureAISearchVectorStoreclass, and subsequently within theVectorStoreIndexand the retriever [1][2].To continue talking to Dosu, mention @dosu.