langchain: ConversationalRetrievalChain doesn't return score with sources

@hwchase17 @agola11 @vowelparrot

Should we add a flag to ConversationalRetrievalChain like similarity = True to get similarity scores from the chain? I can raise a PR, can you assign me this issue?

Here’s my hacky solution. It’s a retriever that saves the score as part of the documents metadata. I just took the original _get_relevant_documents and rewrote the part @khaledalarja pointed us at in his comment.

class MyVectorStoreRetriever(VectorStoreRetriever):
    # See https://github.com/langchain-ai/langchain/blob/61dd92f8215daef3d9cf1734b0d1f8c70c1571c3/libs/langchain/langchain/vectorstores/base.py#L500
    def _get_relevant_documents(
        self, query: str, *, run_manager: CallbackManagerForRetrieverRun
    ) -> List[Document]:
        docs_and_similarities = (
            self.vectorstore.similarity_search_with_relevance_scores(
                query, **self.search_kwargs
            )
        )

        # Make the score part of the document metadata
        for doc, similarity in docs_and_similarities:
            doc.metadata["score"] = similarity

        docs = [doc for doc, _ in docs_and_similarities]
        return docs

Instead of doing vectordb.as_retriever I instantiate the retriever by doing:

retriever = MyVectorStoreRetriever(
   vectorstore=vectordb,
   search_type="similarity_score_threshold",
   search_kwargs={"score_threshold": similarity_threshold, "k": 3},
)

Good luck 😃

This issue should be reopened, the issue is still not solved, the PR #5655 is not related to the behavior of ConversationalRetrievalChain, it’s only specific for Chroma, but here it’s the problem, scores are neglected with the _

@KeshavSingh29 great question - the short answer is we’re working with the maintainers of the vector stores towards that goal.

The longer answer is that each of the vector stores use different distance or similarity functions to compute scores (that also frequently are sensitive to the embeddings you’re using). Sometimes it’s a distance metric (lower number means more similar), while (less often) it’s a similarity metric (higher number is more similar), and the results are usually unbounded.

We could add a default implementation that tries to capture the most common use cases but we have refrained from trying to put a ‘one size fits all’ normalization function in the base class to avoid providing erroneous results

Probably a dumb question, but what version of which package do I need to have for this to work? I’ve just done a pip install yesterday and it’s still not returning scores.

Took me a bit to find the correct imports :

from langchain.schema.vectorstore import VectorStoreRetriever
from langchain.callbacks.manager import CallbackManagerForRetrieverRun
from langchain.schema.document import Document
from typing import List
...

is this resolved? I am trying to get a score when using ConversationalRetrivalChain with the FAISS vector store.

@vowelparrot, I am using ConversationalRetrievalChain to chat over multiple files (some of them are PDF, some docx, txt,…). So for me is very important to also return the source, so I easily know from which file the answer came. This however works great if I ask the model question, which is related to the content from some of the documents. If question is very different, like ‘What is the distance from moon to mars?’ or ‘Write me a song about …’ the model still gives me source, but that source is the first embedding in database. This technically is not ‘wrong’, because this still is source but the distance from vector is very very large.

Here score comes in to make wonders. I could easily than filter the sources before displaying them, if the score is low.

I hope this will be a good enough answer for you to raise the PR for @AvikantSrivastava to integrate this funcionality.

Wish all of you a nice day.