pyterrier: RM3 does not add additional terms to the query for very small corpora

Dear all,

Thank you very much for your efforts with PyTerrier, it is a really awesome framework! I currently want to create small Hello World examples to showcase some concepts for our IR course, and it looks to me like RM3 does not behave as I would expect in my example, because RM3 does not expand the query (maybe I used it wrong, or there are some caveats because my example is so small). This is not urgent, as it works with Bo1QueryExpansion and KLQueryExpansion.

Describe the bug RM3 does not add additional terms to the query in the hello world example described below. E.g., For the same relevance feedback document, RM3 expands the query dog to applypipeline:off dog^0.600000024 whereas Bo1 expanded the query dog to applypipeline:off dog^2.000000000 colli^1.000000000 3^1.000000000 border^1.000000000 shepherd^1.000000000 1^1.000000000 type^1.000000000 german^1.000000000 2^1.000000000 poodl^0.805050646.

To Reproduce

I have prepared a Colab notebook that showcases the problem.

Steps to reproduce the behavior:

  1. Which index
documents = [
    {'docno': 'd1', 'text': 'The Golden Retriever is a Scottish breed of medium size.'},
    {'docno': 'd2', 'text': 'Intelligent types of dogs are: (1) Border Collies, (2) Poodles, and (3) German Shepherds.'},
    {'docno': 'd3', 'text': 'Poodles are a highly intelligent, energetic, and sociable.'},
    {'docno': 'd4', 'text': 'The European Shorthair is medium-sized to large cat with a well-muscled chest.'},
    {'docno': 'd5', 'text': 'The domestic canary is a small songbird.'}
]

if not pt.started():
    pt.init(boot_packages=["com.github.terrierteam:terrier-prf:-SNAPSHOT"])

indexer = pt.IterDictIndexer("/tmp/index", overwrite=True, blocks=True)
index_ref = indexer.index(documents)
index = pt.IndexFactory.of(index_ref)
  1. Which topics
topic = pd.DataFrame([{'qid': '1', 'query': 'dog'}])
  1. Which pipeline
explicit_relevance_feedback = pt.Transformer.from_df(pd.DataFrame([{'query': 'dog', 'qid': '1', 'docno': 'd2'}]))

bo1_expansion = explicit_relevance_feedback >> pt.rewrite.Bo1QueryExpansion(index)
rm3_expansion = explicit_relevance_feedback >> pt.rewrite.RM3(index)
  1. Output

Screenshot_20231022_104422

Expected behavior

I would expect that RM3 also adds terms like colli, shepherd, poodl, etc to the query.

Documentation and Issues

Thanks in advance!

Best regards,

Maik

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

Now fixed, thanks Maik!

Thanks @mam10eks, we’ll give a look at the code in the PR

Hi, I have not yet started with this, but at the beginning of next week I have time to look into this 😃