llama_index: [Bug]: BM25Retriever cannot work on chinese
Bug Description
BM25Retriever cannot work on chinese.
Version
main
Steps to Reproduce
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.response.notebook_utils import display_source_node
documents = [Document(text="床前明月光"),
Document(text="疑是地上霜"),
Document(text="举头望明月"),
Document(text="低头思故乡")]
splitter = SentenceSplitter(chunk_size=1024)
nodes = splitter.get_nodes_from_documents(documents)
retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=2)
nodes = retriever.retrieve("故乡")
for node in nodes:
display_source_node(node)
Relevant Logs/Tracbacks
output:
Node ID: d3d59a82-e489-47da-8229-a430c7f58c00 Similarity: 0.0 Text: 床前明月光
Node ID: 6105e407-408e-48a5-af44-a82515babb56 Similarity: 0.0 Text: 疑是地上霜
Node ID: fd059a68-d496-4a61-847f-c96e108a69b2 Similarity: 0.0 Text: 举头望明月
Node ID: f19761bc-d381-4679-bc77-f32cbad57db5 Similarity: 0.0 Text: 低头思故乡
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 20 (5 by maintainers)
Thanks for your suggestion. Yes, I agree with you that BM25Retriever and BM25SRetriever should be different things. And I made myself a Chinese vesion of BM25Retriver using bm25s finally.
And this is my poc code if anyone needed:
@lifu963 你好! 我是 Dosu。作为一个机器人,我可以帮你解决bug,回答问题,还可以帮你成为贡献者。在等待人类维护者的时候,我会全程帮助你。
To resolve the issue with the
BM25Retrievernot correctly processing Chinese text, you need to modify the tokenizer to handle Chinese text appropriately. The current tokenizer is designed for English text, which is not suitable for Chinese.Here is how you can modify the tokenizer to handle Chinese text:
Install the
jiebalibrary, which is a popular Chinese text segmentation library:Define a new tokenizer function for Chinese text:
Use this new tokenizer when initializing the
BM25Retriever:By using
chinese_tokenizer, theBM25Retrieverwill be able to correctly process Chinese text and compute meaningful similarity scores [1][2].To continue talking to Dosu, mention @dosu.
based on the code of stg609, I think make ChineseBM25Retriever inherit from BM25Retriever is more appropriate cause it can use the class method
persistdefined in BM25Retriever.this interests me. checked bm25s and a new BM25Retriever. my current understanding is below:
my summary:
also, about stemmer.
bm25s uses pystemmer as optional and it does not support Japanese. In japanese, stemming is too simple to use and we normally use normalization which is more like lemmatization instead of stemming, and it tightly connects with tokenization. so if I implement, I will implement some kind of stemming, normalization or lemmatization inside of tokenizer, not use stemmer option.
so as current, I will use old version of BM25Retriever and pass my tokenizer.
I have not tested new BM25Retriever. so if my understanding is wrong, sorry in advanced.
In the version 0.10.57, the tokenizer option is marked as deprecated and suggest to use a stemmer instead. How can I make use of stemmer for Chinese?
这是来自QQ邮箱的假期自动回复邮件。 您好,我最近正在休假中,无法亲自回复您的邮件。我将在假期结束后,尽快给您回复。
这是来自QQ邮箱的假期自动回复邮件。 您好,我最近正在休假中,无法亲自回复您的邮件。我将在假期结束后,尽快给您回复。