FlashRAG: Problem in replicating results in Wiki Data

Hi. As a first step we were able to run the example code. However, now we are moving to the wiki data and trying to run the wiki data-based experiments. Towards this, we have downloaded the corpus and flat index from huggingface (https://huggingface.co/datasets/ignore/FlashRAG_datasets/tree/main/retrieval-corpus) .

As we are modifying the given simple_pipeline.py to fit to the wiki data, we are facing some trouble. Can you please help us out?

For your reference, we just did the following changes:

In the simple pipeline we just changed the corpus and index path to the downloaded corpus and index:

config_dict = { ‘data_dir’: ‘dataset/’, ‘index_path’: ‘indexes/wiki18_100w_e5_flat.index’, ‘corpus_path’: ‘indexes/wiki18_100w.jsonl’, ‘model2path’: {‘e5’: args.retriever_path, ‘llama3-8B-instruct’: args.model_path}, ‘generator_model’: ‘llama3-8B-instruct’, ‘retrieval_method’: ‘e5’, ‘metrics’: [‘em’,‘f1’,‘sub_em’], ‘retrieval_topk’: 1, ‘save_intermediate_data’: True }

We are facing a weird error: Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Find reference in template Find question in template Traceback (most recent call last): File “/data/wikimedia/FlashRAG/FlashRAG/examples/quick_start/wiki_pipeline.py”, line 35, in <module> pipeline = SequentialPipeline(config, prompt_template=prompt_templete) File “/data/wikimedia/FlashRAG/FlashRAG/flashrag/pipeline/pipeline.py”, line 54, in init self.retriever = get_retriever(config) File “/data/wikimedia/FlashRAG/FlashRAG/flashrag/utils/utils.py”, line 76, in get_retriever return getattr( File “/data/wikimedia/FlashRAG/FlashRAG/flashrag/retriever/retriever.py”, line 230, in init self.index = faiss.read_index(self.index_path) File “/csmishra/softwares/anaconda/envs/flashRAG/lib/python3.9/site-packages/faiss/swigfaiss_avx2.py”, line 12365, in read_index return _swigfaiss_avx2.read_index(args) RuntimeError: Error in faiss::Index faiss::read_index(faiss::IOReader*, int) at /home/circleci/miniconda/conda-bld/faiss-pkg_1709244517602/work/faiss/impl/index_read.cpp:1053: Index type 0x61746164 (“data”) not recognized

Can someone please help us out here?

–Regards Cyan

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

You can just add gpu_id into your config_dict and set the correct value. It will replace the default value.

Let me try a fresh run on a different machine in my morning time. If I could not get it fixed, I’ll get back to you. For now, I belive, it is the problem with the hardware environment. Thanks for all the help. I’ll close this issue now and open a new one if needed.

I must say - awesome help and very quick response… 👍 💯