pyterrier: RM3 does not add additional terms to the query for very small corpora
Dear all,
Thank you very much for your efforts with PyTerrier, it is a really awesome framework! I currently want to create small Hello World examples to showcase some concepts for our IR course, and it looks to me like RM3 does not behave as I would expect in my example, because RM3 does not expand the query (maybe I used it wrong, or there are some caveats because my example is so small). This is not urgent, as it works with Bo1QueryExpansion and KLQueryExpansion.
Describe the bug
RM3 does not add additional terms to the query in the hello world example described below.
E.g., For the same relevance feedback document, RM3 expands the query dog to applypipeline:off dog^0.600000024 whereas Bo1 expanded the query dog to applypipeline:off dog^2.000000000 colli^1.000000000 3^1.000000000 border^1.000000000 shepherd^1.000000000 1^1.000000000 type^1.000000000 german^1.000000000 2^1.000000000 poodl^0.805050646.
To Reproduce
I have prepared a Colab notebook that showcases the problem.
Steps to reproduce the behavior:
- Which index
documents = [
{'docno': 'd1', 'text': 'The Golden Retriever is a Scottish breed of medium size.'},
{'docno': 'd2', 'text': 'Intelligent types of dogs are: (1) Border Collies, (2) Poodles, and (3) German Shepherds.'},
{'docno': 'd3', 'text': 'Poodles are a highly intelligent, energetic, and sociable.'},
{'docno': 'd4', 'text': 'The European Shorthair is medium-sized to large cat with a well-muscled chest.'},
{'docno': 'd5', 'text': 'The domestic canary is a small songbird.'}
]
if not pt.started():
pt.init(boot_packages=["com.github.terrierteam:terrier-prf:-SNAPSHOT"])
indexer = pt.IterDictIndexer("/tmp/index", overwrite=True, blocks=True)
index_ref = indexer.index(documents)
index = pt.IndexFactory.of(index_ref)
- Which topics
topic = pd.DataFrame([{'qid': '1', 'query': 'dog'}])
- Which pipeline
explicit_relevance_feedback = pt.Transformer.from_df(pd.DataFrame([{'query': 'dog', 'qid': '1', 'docno': 'd2'}]))
bo1_expansion = explicit_relevance_feedback >> pt.rewrite.Bo1QueryExpansion(index)
rm3_expansion = explicit_relevance_feedback >> pt.rewrite.RM3(index)
- Output
Expected behavior
I would expect that RM3 also adds terms like colli, shepherd, poodl, etc to the query.
Documentation and Issues
- I have checked the PyTerrier documentation for relevant content
- I have checked for previous relevant PyTerrier issues
Thanks in advance!
Best regards,
Maik
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (18 by maintainers)
Commits related to this issue
- Start to prepare unit tests for RM3 to ensure the changes in ticket #407 behave as expected — committed to mam10eks/pyterrier by mam10eks 2 years ago
- Add unit tests that show the failure case for #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- Add unit tests for non-rm3 query expansion methods to ensure that fixing #407 will not change their behaviour — committed to mam10eks/pyterrier by mam10eks 2 years ago
- inject retrieval scores for bm25 #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- Introduce requires_scores for QueryExpansion transformers like RM3 #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- Introduce requires_scores for QueryExpansion transformers like RM3 #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- Add RM3 unit tests into the github push workflow #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- change default scores to fload as before #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- cache topics_and_res for qid and remove unneded score dictionary #407 — committed to mam10eks/pyterrier by mam10eks 2 years ago
- Incorporate Retrieval Scores into RM3 (#453), addresses #407 Co-Authored by Maik Frobe @mam10eks — committed to terrier-org/pyterrier by mam10eks 2 years ago
Now fixed, thanks Maik!
Thanks @mam10eks, we’ll give a look at the code in the PR
Hi, I have not yet started with this, but at the beginning of next week I have time to look into this 😃