llama_index: [Bug]: Document docstore/data not being created on occassion
Bug Description
Occasionally the document docstore/data is not being created (KVDocumentStore) when add_documents is called
However docstore/metadata is consistently being created.
This is leading to an issue when aget_document is called and the document cannot be found because it does not exist for the /data collection
Version
0.10.67.post1
Steps to Reproduce
I’ve been unable to intentionally reproduce this. However we have several thousand occurrences of the docstore/data document not being created when add_documents is called.
99% of the time it works and I can’t determine what the cause of failure to create this is.
Relevant Logs/Tracbacks
async def insert_faq(self: Self, params: FAQCreateParams) -> FAQ:
if "uid" in params:
document_exists = await self.docstore.adocument_exists(params["uid"])
if document_exists:
raise ValueError(f"[FAQAlreadyExists] uid: {params['uid']}")
faq = FAQ.from_params(**params.model_dump())
faq_document = faq.to_document()
self.index.insert(faq_document)
await self.docstore.async_add_documents([faq_document])
return faq
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 21
Hmm,
insert()is sync, and therefore blocking, so I don’t see that being an issue (unless you are multithreading stuff)I don’t think that issue is related, that’s more so talking about the lack of an async insert function on the index class
ah ok, so postgres
I’m far from an expert on Postgres, but maybe you can spot something in this code that might cause this to happen? https://github.com/run-llama/llama_index/blob/d658f4f427b3e2910ab44aa579bae21aacfc9c01/llama-index-integrations/storage/kvstore/llama-index-storage-kvstore-postgres/llama_index/storage/kvstore/postgres/base.py#L289
we thought perhaps this but its never getting created - the issues were seeing data back to several months ago