ragas: error: RagasEvaluatorChain with non-openai models

Describe the bug

Cannot get the evaluation with Langchain QA Chains using non-openai models.

Ragas version: 0.0.21 Python version: 3.10.0

Code to Reproduce

# list of metrics we're going to use
metrics = [
    answer_relevancy,
    faithfulness,
    context_precision,
    context_recall,
]

for m in metrics:
    m.__setattr__("llm", bedrock_llm)

answer_relevancy_chain = RagasEvaluatorChain(metric=metrics[0])
eval_result = answer_relevancy_chain(result)
eval_result["answer_relevancy_score"]

Error trace

Traceback (most recent call last):
>>> eval_result = answer_relevancy_chain(result)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/python3.10/site-packages/langchain/chains/base.py", line 312, in __call__
    raise e
  File "/python3.10/site-packages/langchain/chains/base.py", line 306, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/python3.10/site-packages/ragas/langchain/evalchain.py", line 70, in _call
    score = self.metric.score_single(
  File "/python3.10/site-packages/ragas/metrics/base.py", line 101, in score_single
    score = self._score_batch(
  File "/python3.10/site-packages/ragas/metrics/answer_relevance.py", line 87, in _score_batch
    results = self.llm.generate(
  File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 370, in generate
    raise e
  File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 360, in generate
    self._generate_with_cache(
  File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 514, in _generate_with_cache
    return self._generate(
  File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 99, in _generate
    prompt = ChatPromptAdapter.convert_messages_to_prompt(
  File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 30, in convert_messages_to_prompt
    prompt = convert_messages_to_prompt_anthropic(messages=messages)
  File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 64, in convert_messages_to_prompt_anthropic
    text = "".join(
  File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 65, in <genexpr>
    _convert_one_message_to_text(message, human_prompt, ai_prompt)
  File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 31, in _convert_one_message_to_text
    content = cast(str, message.content)
AttributeError: 'tuple' object has no attribute 'content'

Expected behavior It should execute all ragas metrics without raising an error.

About this issue

  • Original URL
  • State: open
  • Created 7 months ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Good point, sorry for misunderstanding, I was assuming that it was just the base LLM (Claude in my case) that was returning the error. Definitely makes sense now that it would come up when we ask it to generate embeddings. I will try Cohere as you suggested.