tevatron: unable to reproduce repllama performance
Hello, thanks for sharing great work!
I tried trying repllama myself with the repllama branch but failed to reproduce the numbers. Could you check whether any of my hyperparameters are wrong? I add the training script underneath I am currently running on 8 A100 and the result I got is NDCG@10: 0.3959, NDCG@100: 0.4515 When I download the released model from hf I get the number in the paper, which I assume the issue is from the training not evaluation.
Thanks 😃
deepspeed --master_port 40000 train.py \
--deepspeed "ds_config.json" \
--output_dir "model_repllama_lora_train.7b.re" \
--model_name_or_path "meta-llama/Llama-2-7b-hf" \
--save_steps 500 \
--dataset_name "Tevatron/msmarco-passage" \
--bf16 \
--per_device_train_batch_size 16 \
--gradient_accumulation_steps 1 \
--gradient_checkpointing \
--train_n_passages 16 \
--learning_rate 1e-4 \
--q_max_len 32 \
--p_max_len 196 \
--num_train_epochs 1 \
--logging_steps 10 \
--overwrite_output_dir \
--dataset_proc_num 32 \
--negatives_x_device \
--warmup_steps 100 \
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (7 by maintainers)
Hi @orionw, my reproduce with lora_r=32, others keeps same, gives: dev mrr@10: 41.6 dl19 ndcg@10: 74.6 dl20 ndcg@10: 71.4
dev/dl19 a bit higher and dl20 a bit lower than original experiments.
one difference I observed is the lora_r parameters, it was set to 32 in original experiment, now the default one is 8. I am seeing if this would affect trec dl results.
I see. I’ll schedule a training to see how it goes on my end…
Hi @MXueguang @amy-hyunji, Yes I tried a similar training config as @amy-hyunji which set gradient_accumulation_steps to 1 and it won’t work well…, adding back gradient_accumulation_steps to 4 will reproduce the results.
Note gradient_accumulation_steps will affect the number of negatives per training example, For example, if set per_device_train_batch_size 8 and gradient_accumulation_steps 4 with 4 gpus, this will result in a total batch size of 128 and the number of negatives per example is 4 * 8 * 16 = 512. If per_device_train_batch_size to 32, gradient_accumulation_steps 1 with 4 gpus. the batch size is also 128 but the number of negatives will be 4 * 32 * 16 = 2048 (@MXueguang correct me if I’m wrong).
This is odd to me as well as the experience from the literature is more negative is better…
for reproducing repllama training, I’d suggest using the main branch and using the latest code
you can either use the mistral as initialization for a little higher effectiveness or use llama2 for reproducing.
Notice that the training data
Tevatron/msmarco-passage-augis the data used to train repllama