vertex-ai-samples: Vertex AI pipeline - IndexError: Invalid key: 0 is out of bounds for size 0
Expected Behavior
The fine-tuning of the foundation model should complete without any issues.
Actual Behavior
The fine-tuning step gets terminated. The details provided below:
Training framework - Google collab
Model used - Llama2-7B
Fine-tuning method - PEFT
Number of samples in Training Set - 100
Number of samples in Eval Set - 20
Format of the training data - jsonl
Example sample is given below -
{"text": "### Human: What is arithmatic mean? ### Assistant: The arithmetic mean, or simply the mean, is the average of a set of numbers obtained by adding them up and dividing by the total count of numbers."}
{"text": "### Human: What is geometric mean? ### Assistant: The geometric mean is a measure of central tendency calculated by multiplying all values in a dataset and then taking the nth root of the product, where n is the total number of values."}
Vertex pipeline parameters :
pipeline_parameters = {
"base_model": base_model,
"dataset_name": dataset_name,
"prediction_accelerator_type": prediction_accelerator_type,
"training_accelerator_type": training_accelerator_type,
"training_precision_mode": training_precision_mode,
"training_lora_rank": 16,
"training_lora_alpha": 32,
"training_lora_dropout": 0.05,
"training_steps": 20,
"training_warmup_steps": 10,
"training_learning_rate": 2e-4,
"evaluation_steps": 10,
"evaluation_limit": 1,
}
When I execute the training process, I get the below error:
raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 0 is out of bounds for size 0
Can you please help in understanding the below question?
- Is the format of training data correct ? I used the format which was given as default example in Collab notebook, you can find the dataset here
- Is the number of samples too less ?
- Is there anything I am missing here ?
Steps to Reproduce the Problem
Specifications
- Version:
- Platform:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (6 by maintainers)
Yea initially I tried with huggingface dataset and got it working. Later with the same dataset I migrated to Google bucket, it worked as expected.
Thank you,
Kk
@genquan9: can you please assist with this? Thank you.
Hi @gericdong I am using the below notebook
model_garden_pytorch_llama2_peft_finetuning.ipynbThank you,
KK