llama_index: [Bug]: LLMPredictor Cache arguement

Bug Description

I want to add gptcache object in cache to have that in llama index, but now LLM Predictor is not taking up cache arguments, how can we do caching now.

index = GPTVectorStoreIndex.from_documents(
    documents,
    service_context=ServiceContext.from_defaults(
        llm_predictor=LLMPredictor(cache=gptcache_obj)
    ),

cache=gptcache_obj throws error as cache is not a valid argument.

Version

latest

Steps to Reproduce

try to integrate llama index with gpt cache

import hashlib
from gptcache import Cache
from gptcache.adapter.api import init_similar_cache


def get_hashed_name(name):
    return hashlib.sha256(name.encode()).hexdigest()


def init_gptcache(cache_obj: Cache, llm: str):
    hashed_llm = get_hashed_name(llm)
    init_similar_cache(cache_obj=cache_obj, data_dir=f"similar_cache_{hashed_llm}")

gptcache_obj = GPTCache(init_gptcache)

from llama_index import (
    GPTVectorStoreIndex,
    ServiceContext,
    LLMPredictor,
    SimpleWebPageReader,
)

loader = SimpleWebPageReader(html_to_text=True)
documents = loader.load_data(urls=["https://milvus.io/docs/overview.md"])

index = GPTVectorStoreIndex.from_documents(
    documents,
    service_context=ServiceContext.from_defaults(
        llm_predictor=LLMPredictor(cache=gptcache_obj)
    ),
)
query_engine = index.as_query_engine()

Relevant Logs/Tracbacks

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15

Most upvoted comments

🤖

Hello,

Thank you for reaching out. It seems like you’re trying to integrate GPTCache with LlamaIndex. However, as you’ve noticed, the LLMPredictor class does not currently accept a cache argument.

Based on similar issues that have been resolved in the past, it appears that caching can be achieved by setting the cache on the langchain module if you are using langchain LLM classes. Here is a sample code snippet from a previous issue:

import langchain
from langchain.cache import InMemoryCache

langchain.llm_cache = InMemoryCache()

from langchain.chat_models import ChatOpenAI
from llama_index import ListIndex, Document, ServiceContext

llm = ChatOpenAI(model_name="gpt-3.5-turbo")
service_context = ServiceContext.from_defaults(llm=llm)
index = ListIndex.from_documents([Document("Hello World")], service_context=service_context)
index.as_query_engine().query("Hello?")
index.as_query_engine().query("Hello?")

In this example, the second query will return instantly due to the caching mechanism. This feature should be added to the native LLMs at some point too.

Please note that this solution is based on the langchain module, and it might not be directly applicable if you’re not using langchain LLM classes. If you’re not using langchain, could you please provide more information about your setup so I can assist you better?

I hope this helps. If you have any other questions or need further clarification, feel free to ask.

Best regards, Dosu

Sources


This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

Well i think best would be if we can have cache arguments, the way it is mentioned in GPT cache documentation. is there any plan to have this in future releases?

🤖

Based on the current architecture of the LlamaIndex data framework, caching is handled by storing parent nodes in a dictionary called parent_nodes. This dictionary acts as a cache, storing parent nodes as they are retrieved from the document store. This approach helps to avoid unnecessary database calls by reusing previously fetched parent nodes, thus improving the performance of the data framework.

If you want to implement caching in your context, you can follow a similar approach. Here’s an example of how you can modify your get_query_engine method to include caching:

class QueryEngineCache:
    def __init__(self):
        self.cache = {}

    def get_query_engine(self, tenant_id, tenant_index, tenant_config, model_name, node_postprocessors, text_qa_template, synthesizer_mode):
        cache_key = (tenant_id, tenant_index, model_name)
        if cache_key in self.cache:
            return self.cache[cache_key]

        query_engine = self.__llama_idx_svc.get_query_engine(tenant_id,
                                                             tenant_index,
                                                             tenant_config,
                                                             model_name,
                                                             node_postprocessors=node_postprocessors,
                                                             text_qa_template=text_qa_template,
                                                             synthesizer_mode=synthesizer_mode)

        if query_engine is None:
            raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Unable to construct Query Engine")

        self.cache[cache_key] = query_engine
        return query_engine

query_engine_cache = QueryEngineCache()

And then you can use the query_engine_cache object to get the query engine:

query_engine = query_engine_cache.get_query_engine(tenant_id,
                                                   tenant_index,
                                                   tenant_config,
                                                   model_name,
                                                   node_postprocessors=node_postprocessors,
                                                   text_qa_template=text_qa_template,
                                                   synthesizer_mode=synthesizer_mode)

response = query_engine.query(query)

This way, if a query engine with the same parameters has been created before, it will be retrieved from the cache instead of being created anew. This can significantly improve the performance of your application, especially when dealing with a large number of queries.

Please note that this is a basic example and might need to be adjusted based on your specific requirements and the overall architecture of your application. For instance, you might need to implement cache eviction strategies if memory usage becomes a concern.

Sources


This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.