ragas: Testset generation error on Bedrock

Describe the bug A clear and concise description of what the bug is.

Ragas version: 0.1.0rc2.dev26+g6a88465 Python version: 3.10.13

Code to Reproduce

###Add custom llms and embeddings
generator_llm = LangchainLLMWrapper(candidate_llm1)
critic_llm = LangchainLLMWrapper(judge_llm)
embeddings_model = LangchainEmbeddingsWrapper(titan_embeddings,RunConfig(max_retries=2, max_wait=30))

Change resulting question type distribution
testset_distribution = {
"simple": 0.25,
"reasoning": 0.5,
"multi_context": 0.0,
"conditional": 0.25,
}

splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = keyphraseExtractor(llm=generator_llm)

docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=embeddings_model,
extractor=keyphrase_extractor,
)
test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=titan_embeddings,
docstore=docstore
)

testset = test_generator.generate_with_langchain_docs(documents=documents,
test_size=10,
distributions=testset_distribution)

Error trace

/ragas/src/ragas/testset/docstore.py:250, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress, desc)
248 for i, n in enumerate(nodes):
249 if i in nodes_to_embed.keys():
--> 250 n.embedding = results[nodes_to_embed[i]]
251 if i in nodes_to_extract.keys():
252 keyphrases = results[nodes_to_extract[i]]

IndexError: list index out of range

Expected behavior A clear and concise description of what you expected to happen.

Additional context langchain documents. Tried with diffrent chunk sizes and overlapps, different embedding configs and different bedrock models(cohere and titan text) I am getting with any model on Bedrock “IndexError: list index out of range” happens exactly at 50% point all the time

About this issue

  • Original URL
  • State: closed
  • Created 5 months ago
  • Comments: 17 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Let me know if you’re working on it so that I can create an issue and assign it to you and make sure non one else is doing the same. @subha-aws

hey @subha-aws can you update to v0.1. I don’t think that fixes this issue but the error trace you gave is a bit misleading I think. Need the full trace for v0.1. If you could pass that that would be really helpful for us to reproduce 😃

Hey @subha-aws , thanks for sharing this bug. I have also noticed this, and will fix is ASAP