llama_index: [Bug]: LLMPredictor Cache arguement
Bug Description
I want to add gptcache object in cache to have that in llama index, but now LLM Predictor is not taking up cache arguments, how can we do caching now.
index = GPTVectorStoreIndex.from_documents(
documents,
service_context=ServiceContext.from_defaults(
llm_predictor=LLMPredictor(cache=gptcache_obj)
),
cache=gptcache_obj throws error as cache is not a valid argument.
Version
latest
Steps to Reproduce
try to integrate llama index with gpt cache
import hashlib
from gptcache import Cache
from gptcache.adapter.api import init_similar_cache
def get_hashed_name(name):
return hashlib.sha256(name.encode()).hexdigest()
def init_gptcache(cache_obj: Cache, llm: str):
hashed_llm = get_hashed_name(llm)
init_similar_cache(cache_obj=cache_obj, data_dir=f"similar_cache_{hashed_llm}")
gptcache_obj = GPTCache(init_gptcache)
from llama_index import (
GPTVectorStoreIndex,
ServiceContext,
LLMPredictor,
SimpleWebPageReader,
)
loader = SimpleWebPageReader(html_to_text=True)
documents = loader.load_data(urls=["https://milvus.io/docs/overview.md"])
index = GPTVectorStoreIndex.from_documents(
documents,
service_context=ServiceContext.from_defaults(
llm_predictor=LLMPredictor(cache=gptcache_obj)
),
)
query_engine = index.as_query_engine()
Relevant Logs/Tracbacks
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15
🤖
Hello,
Thank you for reaching out. It seems like you’re trying to integrate GPTCache with LlamaIndex. However, as you’ve noticed, the
LLMPredictorclass does not currently accept acacheargument.Based on similar issues that have been resolved in the past, it appears that caching can be achieved by setting the cache on the langchain module if you are using langchain LLM classes. Here is a sample code snippet from a previous issue:
In this example, the second query will return instantly due to the caching mechanism. This feature should be added to the native LLMs at some point too.
Please note that this solution is based on the langchain module, and it might not be directly applicable if you’re not using langchain LLM classes. If you’re not using langchain, could you please provide more information about your setup so I can assist you better?
I hope this helps. If you have any other questions or need further clarification, feel free to ask.
Best regards, Dosu
Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
Well i think best would be if we can have cache arguments, the way it is mentioned in GPT cache documentation. is there any plan to have this in future releases?
🤖
Based on the current architecture of the LlamaIndex data framework, caching is handled by storing parent nodes in a dictionary called
parent_nodes. This dictionary acts as a cache, storing parent nodes as they are retrieved from the document store. This approach helps to avoid unnecessary database calls by reusing previously fetched parent nodes, thus improving the performance of the data framework.If you want to implement caching in your context, you can follow a similar approach. Here’s an example of how you can modify your
get_query_enginemethod to include caching:And then you can use the
query_engine_cacheobject to get the query engine:This way, if a query engine with the same parameters has been created before, it will be retrieved from the cache instead of being created anew. This can significantly improve the performance of your application, especially when dealing with a large number of queries.
Please note that this is a basic example and might need to be adjusted based on your specific requirements and the overall architecture of your application. For instance, you might need to implement cache eviction strategies if memory usage becomes a concern.
Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.