ragas: Testset generation error on Bedrock
Describe the bug A clear and concise description of what the bug is.
Ragas version: 0.1.0rc2.dev26+g6a88465 Python version: 3.10.13
Code to Reproduce
###Add custom llms and embeddings
generator_llm = LangchainLLMWrapper(candidate_llm1)
critic_llm = LangchainLLMWrapper(judge_llm)
embeddings_model = LangchainEmbeddingsWrapper(titan_embeddings,RunConfig(max_retries=2, max_wait=30))
Change resulting question type distribution
testset_distribution = {
"simple": 0.25,
"reasoning": 0.5,
"multi_context": 0.0,
"conditional": 0.25,
}
splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = keyphraseExtractor(llm=generator_llm)
docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=embeddings_model,
extractor=keyphrase_extractor,
)
test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=titan_embeddings,
docstore=docstore
)
testset = test_generator.generate_with_langchain_docs(documents=documents,
test_size=10,
distributions=testset_distribution)
Error trace
/ragas/src/ragas/testset/docstore.py:250, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress, desc)
248 for i, n in enumerate(nodes):
249 if i in nodes_to_embed.keys():
--> 250 n.embedding = results[nodes_to_embed[i]]
251 if i in nodes_to_extract.keys():
252 keyphrases = results[nodes_to_extract[i]]
IndexError: list index out of range
Expected behavior A clear and concise description of what you expected to happen.
Additional context langchain documents. Tried with diffrent chunk sizes and overlapps, different embedding configs and different bedrock models(cohere and titan text) I am getting with any model on Bedrock “IndexError: list index out of range” happens exactly at 50% point all the time
About this issue
- Original URL
- State: closed
- Created 5 months ago
- Comments: 17 (8 by maintainers)
Commits related to this issue
- fix: ensure dict type (#578) fixes: #563 — committed to explodinggradients/ragas by shahules786 5 months ago
Let me know if you’re working on it so that I can create an issue and assign it to you and make sure non one else is doing the same. @subha-aws
hey @subha-aws can you update to v0.1. I don’t think that fixes this issue but the error trace you gave is a bit misleading I think. Need the full trace for v0.1. If you could pass that that would be really helpful for us to reproduce 😃
Hey @subha-aws , thanks for sharing this bug. I have also noticed this, and will fix is ASAP