llama_index: [Bug]: BM25Retriever cannot work on chinese

Bug Description

BM25Retriever cannot work on chinese.

Version

main

Steps to Reproduce

from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.response.notebook_utils import display_source_node

documents = [Document(text="床前明月光"),
             Document(text="疑是地上霜"),
             Document(text="举头望明月"),
             Document(text="低头思故乡")]

splitter = SentenceSplitter(chunk_size=1024)
nodes = splitter.get_nodes_from_documents(documents)
retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=2)

nodes = retriever.retrieve("故乡")
for node in nodes:
    display_source_node(node)

Relevant Logs/Tracbacks

output:

Node ID: d3d59a82-e489-47da-8229-a430c7f58c00 Similarity: 0.0 Text: 床前明月光
Node ID: 6105e407-408e-48a5-af44-a82515babb56 Similarity: 0.0 Text: 疑是地上霜
Node ID: fd059a68-d496-4a61-847f-c96e108a69b2 Similarity: 0.0 Text: 举头望明月
Node ID: f19761bc-d381-4679-bc77-f32cbad57db5 Similarity: 0.0 Text: 低头思故乡

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 20 (5 by maintainers)

Most upvoted comments

Thanks for your suggestion. Yes, I agree with you that BM25Retriever and BM25SRetriever should be different things. And I made myself a Chinese vesion of BM25Retriver using bm25s finally.

And this is my poc code if anyone needed:

import json
import logging
import os

from typing import Any, Callable, Dict, List, Optional, cast
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.base.base_retriever import BaseRetriever
from llama_index.core.callbacks.base import CallbackManager
from llama_index.core.constants import DEFAULT_SIMILARITY_TOP_K
from llama_index.core.indices.vector_store.base import VectorStoreIndex
from llama_index.core.schema import BaseNode, IndexNode, NodeWithScore, QueryBundle
from llama_index.core.storage.docstore.types import BaseDocumentStore
from llama_index.core.vector_stores.utils import (
    node_to_metadata_dict,
    metadata_dict_to_node,
)

import bm25s
import Stemmer
import itertools
import jieba

logger = logging.getLogger(__name__)

DEFAULT_PERSIST_ARGS = {"similarity_top_k": "similarity_top_k", "_verbose": "verbose"}

DEFAULT_PERSIST_FILENAME = "retriever.json"


class ChineseBM25Retriever(BaseRetriever):
    """A BM25 retriever that uses the BM25 algorithm to retrieve nodes.

    Args:
        nodes (List[BaseNode], optional):
            The nodes to index. If not provided, an existing BM25 object must be passed.
        similarity_top_k (int, optional):
            The number of results to return. Defaults to DEFAULT_SIMILARITY_TOP_K.
        callback_manager (CallbackManager, optional):
            The callback manager to use. Defaults to None.
        objects (List[IndexNode], optional):
            The objects to retrieve. Defaults to None.
        object_map (dict, optional):
            A map of object IDs to nodes. Defaults to None.
        verbose (bool, optional):
            Whether to show progress. Defaults to False.
    """


    def _chinese_tokenizer(self, texts: List[str]) -> tuple[str]:
        # Use jieba to segment Chinese text
        rslts= tuple(itertools.chain.from_iterable(jieba.cut(text) for text in texts))
        return rslts
    
    def __init__(
        self,
        nodes: Optional[List[BaseNode]] = None,               
        similarity_top_k: int = DEFAULT_SIMILARITY_TOP_K,
        callback_manager: Optional[CallbackManager] = None,
        objects: Optional[List[IndexNode]] = None,
        object_map: Optional[dict] = None,
        verbose: bool = False,
    ) -> None:
        
        self.similarity_top_k = similarity_top_k

        with open(r'stopwords.txt', encoding='utf-8') as f:
            con = f.readlines()
            stop_words = set() 
            for i in con:
                i = i.rstrip("\n")  
                stop_words.add(i)
        self.stop_words = stop_words

        # for some terms not included in jieba
        with open(r'cn_dict.txt', encoding='utf-8') as f:
            words = f.readlines()
            for word in words:
                jieba.add_word(word)

        corpus_tokens = [
            [word for word in jieba.cut_for_search(node.get_content()) if word not in stop_words and word.strip('\n')]                                  
            for node in nodes
        ]
        self.bm25 = bm25s.BM25()
        from llama_index.core.vector_stores.utils import (
            node_to_metadata_dict
        )
        
        corpus = [node_to_metadata_dict(node) for node in nodes]
        self.bm25.corpus = corpus
        self.bm25.index(corpus_tokens, show_progress=True)

        super().__init__(
            callback_manager=callback_manager,
            object_map=object_map,
            objects=objects,
            verbose=verbose,
        )
    
    def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
        query = query_bundle.query_str

        tokenized_query = [[word for word in jieba.cut_for_search(query) if word not in self.stop_words]]
        
        indexes, scores = self.bm25.retrieve(
            tokenized_query, k=self.similarity_top_k, show_progress=self._verbose
        )

        # batched, but only one query
        indexes = indexes[0]
        scores = scores[0]

        nodes: List[NodeWithScore] = []
        for idx, score in zip(indexes, scores):
            # idx can be an int or a dict of the node
            if isinstance(idx, dict):
                node = metadata_dict_to_node(idx)
            else:
                node_dict = self.corpus[int(idx)]
                node = metadata_dict_to_node(node_dict)
            nodes.append(NodeWithScore(node=node, score=float(score)))

        return nodes

@lifu963 你好! 我是 Dosu。作为一个机器人,我可以帮你解决bug,回答问题,还可以帮你成为贡献者。在等待人类维护者的时候,我会全程帮助你。

To resolve the issue with the BM25Retriever not correctly processing Chinese text, you need to modify the tokenizer to handle Chinese text appropriately. The current tokenizer is designed for English text, which is not suitable for Chinese.

Here is how you can modify the tokenizer to handle Chinese text:

  1. Install the jieba library, which is a popular Chinese text segmentation library:

    pip install jieba
    
  2. Define a new tokenizer function for Chinese text:

    import jieba
    from typing import List
    
    def chinese_tokenizer(text: str) -> List[str]:
        # Use jieba to segment Chinese text
        return list(jieba.cut(text))
    
  3. Use this new tokenizer when initializing the BM25Retriever:

    from llama_index.retrievers.bm25 import BM25Retriever
    from llama_index.core import Document
    from llama_index.core.node_parser import SentenceSplitter
    from llama_index.core.response.notebook_utils import display_source_node
    
    documents = [Document(text="床前明月光"),
                 Document(text="疑是地上霜"),
                 Document(text="举头望明月"),
                 Document(text="低头思故乡")]
    
    splitter = SentenceSplitter(chunk_size=1024)
    nodes = splitter.get_nodes_from_documents(documents)
    
    retriever = BM25Retriever.from_defaults(
        nodes=nodes,
        similarity_top_k=2,
        tokenizer=chinese_tokenizer
    )
    
    nodes = retriever.retrieve("故乡")
    for node in nodes:
        display_source_node(node)
    

By using chinese_tokenizer, the BM25Retriever will be able to correctly process Chinese text and compute meaningful similarity scores [1][2].

To continue talking to Dosu, mention @dosu.

based on the code of stg609, I think make ChineseBM25Retriever inherit from BM25Retriever is more appropriate cause it can use the class method persist defined in BM25Retriever.

import json
import logging
import os

from typing import Any, Callable, Dict, List, Optional, cast
from llama_index.core.base.base_retriever import BaseRetriever
from llama_index.core.callbacks.base import CallbackManager
from llama_index.core.constants import DEFAULT_SIMILARITY_TOP_K
from llama_index.core.indices.vector_store.base import VectorStoreIndex
from llama_index.core.schema import BaseNode, IndexNode, NodeWithScore, QueryBundle
from llama_index.core.storage.docstore.types import BaseDocumentStore
from llama_index.core.vector_stores.utils import (
    node_to_metadata_dict,
    metadata_dict_to_node,
)
from llama_index.core import Document
import bm25s
import itertools
import jieba
from llama_index.retrievers.bm25 import BM25Retriever

logger = logging.getLogger(__name__)

DEFAULT_PERSIST_ARGS = {"similarity_top_k": "similarity_top_k", "_verbose": "verbose"}

DEFAULT_PERSIST_FILENAME = "retriever.json"

class ChineseBM25Retriever(BM25Retriever):
    """A BM25 retriever that uses the BM25 algorithm to retrieve nodes.

    Args:
        nodes (List[BaseNode], optional):
            The nodes to index. If not provided, an existing BM25 object must be passed.
        similarity_top_k (int, optional):
            The number of results to return. Defaults to DEFAULT_SIMILARITY_TOP_K.
        callback_manager (CallbackManager, optional):
            The callback manager to use. Defaults to None.
        objects (List[IndexNode], optional):
            The objects to retrieve. Defaults to None.
        object_map (dict, optional):
            A map of object IDs to nodes. Defaults to None.
        verbose (bool, optional):
            Whether to show progress. Defaults to False.
    """

    def _chinese_tokenizer(self, texts: List[str]) -> tuple[str]:
        # Use jieba to segment Chinese text
        rslts = tuple(itertools.chain.from_iterable(jieba.cut(text) for text in texts))
        return rslts

    def __init__(
            self,
            nodes: Optional[List[BaseNode]] = None,
            similarity_top_k: int = DEFAULT_SIMILARITY_TOP_K,
            callback_manager: Optional[CallbackManager] = None,
            objects: Optional[List[IndexNode]] = None,
            object_map: Optional[dict] = None,
            verbose: bool = False,
    ) -> None:

        super().__init__(
            nodes=nodes,
            similarity_top_k=similarity_top_k,
            callback_manager=callback_manager,
            objects=objects,
            object_map=object_map,
            verbose=verbose,
        )
        
        # change the stop words for Chinese
        with open(r'stopwords.txt', encoding='utf-8') as f: # here needs to add in the path of chinese stopwords
            con = f.readlines()
            stop_words = set()
            for i in con:
                i = i.rstrip('\n')
                stop_words.add(i)
        self.stop_words = stop_words

        corpus_tokens = [
            [word for word in jieba.cut_for_search(node.get_content()) if word not in stop_words and word.strip('\n')]
            for node in nodes
        ]
        corpus = [node_to_metadata_dict(node) for node in nodes]
        self.bm25.corpus = corpus
        self.bm25.index(corpus_tokens, show_progress=True)
    
    def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
        query = query_bundle.query_str

        tokenized_query = [[word for word in jieba.cut_for_search(query) if word not in self.stop_words]]

        indexes, scores = self.bm25.retrieve(
            tokenized_query, k=self.similarity_top_k, show_progress=self._verbose
        )

        # batched, but only one query
        indexes = indexes[0]
        scores = scores[0]

        nodes: List[NodeWithScore] = []
        for idx, score in zip(indexes, scores):
            # idx can be an int or a dict of the node
            if isinstance(idx, dict):
                node = metadata_dict_to_node(idx)
            else:
                node_dict = self.corpus[int(idx)]
                node = metadata_dict_to_node(node_dict)
            nodes.append(NodeWithScore(node=node, score=float(score)))

        return nodes

In the version 0.10.57, the tokenizer option is marked as deprecated and suggest to use a stemmer instead. How can I make use of stemmer for Chinese?

this interests me. checked bm25s and a new BM25Retriever. my current understanding is below:

  • bm25s has a built-in tokenizer but it does not support Chinese and also Japanese (in my case). so need to implement tokenizer by myself in some ways such as implemet tokenizer outside of bm25s, or making a subclass for bm25s and override tokenize method implemented by myself, etc.
  • llamaindex’s BM25Retriever imports bm25s and calls tokenizer method inside of BM25Retriever, this means other languages which bm25s does not support are not suppoted as default.
  • only thing we can in current implementation, we can pass bm25s.BM25() object which contains an index created using bm25s, not llamaindex.
  • as a result, we don’t use llamaindex for indexing stage, only for querying stage.

my summary:

  • make a subclass for bm25s, implement tokenizer for your langage, and override tokenize method wit it.
  • create an index and a retriever from your documents using bm25s, and pass it to new BM25Retriever.

also, about stemmer.

bm25s uses pystemmer as optional and it does not support Japanese. In japanese, stemming is too simple to use and we normally use normalization which is more like lemmatization instead of stemming, and it tightly connects with tokenization. so if I implement, I will implement some kind of stemming, normalization or lemmatization inside of tokenizer, not use stemmer option.

so as current, I will use old version of BM25Retriever and pass my tokenizer.

pip install llama-index-retrievers-bm25==0.1.5

I have not tested new BM25Retriever. so if my understanding is wrong, sorry in advanced.

In the version 0.10.57, the tokenizer option is marked as deprecated and suggest to use a stemmer instead. How can I make use of stemmer for Chinese?

这是来自QQ邮箱的假期自动回复邮件。   您好,我最近正在休假中,无法亲自回复您的邮件。我将在假期结束后,尽快给您回复。

这是来自QQ邮箱的假期自动回复邮件。   您好,我最近正在休假中,无法亲自回复您的邮件。我将在假期结束后,尽快给您回复。