llama_index: [Bug]: Document docstore/data not being created on occassion

Bug Description

Occasionally the document docstore/data is not being created (KVDocumentStore) when add_documents is called However docstore/metadata is consistently being created.

This is leading to an issue when aget_document is called and the document cannot be found because it does not exist for the /data collection

Version

0.10.67.post1

Steps to Reproduce

I’ve been unable to intentionally reproduce this. However we have several thousand occurrences of the docstore/data document not being created when add_documents is called.

99% of the time it works and I can’t determine what the cause of failure to create this is.

Relevant Logs/Tracbacks

async def insert_faq(self: Self, params: FAQCreateParams) -> FAQ:
        if "uid" in params:
            document_exists = await self.docstore.adocument_exists(params["uid"])
            if document_exists:
                raise ValueError(f"[FAQAlreadyExists] uid: {params['uid']}")

        faq = FAQ.from_params(**params.model_dump())
        faq_document = faq.to_document()

        self.index.insert(faq_document)
        await self.docstore.async_add_documents([faq_document])

        return faq

About this issue

Most upvoted comments

Hmm, insert() is sync, and therefore blocking, so I don’t see that being an issue (unless you are multithreading stuff)

I don’t think that issue is related, that’s more so talking about the lack of an async insert function on the index class

Or there’s some delay in your store, and the user queries before its actually fully inserted/created?

we thought perhaps this but its never getting created - the issues were seeing data back to several months ago