ragas: error: RagasEvaluatorChain with non-openai models
Describe the bug
Cannot get the evaluation with Langchain QA Chains using non-openai models.
Ragas version: 0.0.21 Python version: 3.10.0
Code to Reproduce
# list of metrics we're going to use
metrics = [
answer_relevancy,
faithfulness,
context_precision,
context_recall,
]
for m in metrics:
m.__setattr__("llm", bedrock_llm)
answer_relevancy_chain = RagasEvaluatorChain(metric=metrics[0])
eval_result = answer_relevancy_chain(result)
eval_result["answer_relevancy_score"]
Error trace
Traceback (most recent call last):
>>> eval_result = answer_relevancy_chain(result)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/python3.10/site-packages/langchain/chains/base.py", line 312, in __call__
raise e
File "/python3.10/site-packages/langchain/chains/base.py", line 306, in __call__
self._call(inputs, run_manager=run_manager)
File "/python3.10/site-packages/ragas/langchain/evalchain.py", line 70, in _call
score = self.metric.score_single(
File "/python3.10/site-packages/ragas/metrics/base.py", line 101, in score_single
score = self._score_batch(
File "/python3.10/site-packages/ragas/metrics/answer_relevance.py", line 87, in _score_batch
results = self.llm.generate(
File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 370, in generate
raise e
File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 360, in generate
self._generate_with_cache(
File "/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 514, in _generate_with_cache
return self._generate(
File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 99, in _generate
prompt = ChatPromptAdapter.convert_messages_to_prompt(
File "/python3.10/site-packages/langchain/chat_models/bedrock.py", line 30, in convert_messages_to_prompt
prompt = convert_messages_to_prompt_anthropic(messages=messages)
File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 64, in convert_messages_to_prompt_anthropic
text = "".join(
File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 65, in <genexpr>
_convert_one_message_to_text(message, human_prompt, ai_prompt)
File "/python3.10/site-packages/langchain/chat_models/anthropic.py", line 31, in _convert_one_message_to_text
content = cast(str, message.content)
AttributeError: 'tuple' object has no attribute 'content'
Expected behavior It should execute all ragas metrics without raising an error.
About this issue
- Original URL
- State: open
- Created 7 months ago
- Comments: 16 (9 by maintainers)
Good point, sorry for misunderstanding, I was assuming that it was just the base LLM (Claude in my case) that was returning the error. Definitely makes sense now that it would come up when we ask it to generate embeddings. I will try Cohere as you suggested.