F5-TTS: Fine-tune error for this version
Checks
- This template is only for bug reports, usage problems go with ‘Help Wanted’.
- I have thoroughly reviewed the project documentation but couldn’t find information to solve my problem.
- I have searched for existing issues, including closed ones, and couldn’t find a solution.
- I confirm that I am using English to submit this report in order to facilitate communication.
Environment Details
Docker
Steps to Reproduce
Startup successful, but unable to train.
(f5-tts) root@abd144ff4f98:/github/F5-TTS# f5-tts_finetune-gradio --host 0.0.0.0 Building prefix dict from the default dictionary … Loading model from cache /tmp/jieba.cache Loading model cost 0.402 seconds. Prefix dict has been built successfully. Word segmentation module jieba initialized.
Starting app…
- Running on local URL: http://0.0.0.0:7860
To create a public link, set share=True in launch().
run command :
accelerate launch /github/F5-TTS/src/f5_tts/train/finetune_cli.py --exp_name F5TTS_Base --learning_rate 1e-05 --batch_size_per_gpu 1600 --batch_size_type frame --max_samples 64 --grad_accumulation_steps 1 --max_grad_norm 1 --epochs 5000 --num_warmup_updates 72 --save_per_updates 5000 --keep_last_n_checkpoints 5 --last_per_updates 18 --dataset_name wac --finetune --tokenizer pinyin --log_samples --logger wandb
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 2
More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in --num_processes=1.
--num_machines was set to a value of 1
--mixed_precision was set to a value of 'no'
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
Building prefix dict from the default dictionary …
Loading model from cache /tmp/jieba.cache
Building prefix dict from the default dictionary …
Loading model from cache /tmp/jieba.cache
Word segmentation module jieba initialized.
Word segmentation module jieba initialized.
Loading model cost 0.415 seconds. Prefix dict has been built successfully. Loading model cost 0.417 seconds. Prefix dict has been built successfully.
vocab : 3651
vocoder : vocos
vocab : 3651
vocoder : vocos
wandb: Currently logged in as: aochengwang. Use wandb login --relogin to force relogin
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: - Waiting for wandb.init()…
wandb: \ Waiting for wandb.init()…
wandb: | Waiting for wandb.init()…
wandb: Tracking run with wandb version 0.19.2
wandb: Run data is saved locally in /github/F5-TTS/wandb/run-20250117_154459-s03chtpd
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run F5TTS_Base
wandb: ⭐️ View project at https://wandb.ai/aochengwang/wac
wandb: 🚀 View run at https://wandb.ai/aochengwang/wac/runs/s03chtpd
Using logger: wandb
Loading dataset …
Loading dataset …
Download Vocos from huggingface charactr/vocos-mel-24khz
Download Vocos from huggingface charactr/vocos-mel-24khz
Sorting with sampler… if slow, check whether dataset is provided with duration: 0%| | 0/709 [00:00<?, ?it/s] Sorting with sampler… if slow, check whether dataset is provided with duration: 100%|██████████| 709/709 [00:00<00:00, 1855122.61it/s]
Creating dynamic batches with 1600 audio frames per gpu: 0%| | 0/709 [00:00<?, ?it/s] Creating dynamic batches with 1600 audio frames per gpu: 100%|██████████| 709/709 [00:00<00:00, 2779216.39it/s]
Sorting with sampler… if slow, check whether dataset is provided with duration: 0%| | 0/709 [00:00<?, ?it/s] Sorting with sampler… if slow, check whether dataset is provided with duration: 100%|██████████| 709/709 [00:00<00:00, 1870290.27it/s]
Creating dynamic batches with 1600 audio frames per gpu: 0%| | 0/709 [00:00<?, ?it/s] Creating dynamic batches with 1600 audio frames per gpu: 100%|██████████| 709/709 [00:00<00:00, 2650411.35it/s]
Epoch 1/5000: 0%| | 0/76 [00:00<?, ?update/s][rank0]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) [rank1]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Epoch 1/5000: 1%|▏ | 1/76 [00:02<02:55, 2.34s/update]
✔️ Expected Behavior
No response
❌ Actual Behavior
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (5 by maintainers)
Commits related to this issue
- 0.5.0 fix grad_accum bug from 0.4.0, #715 #728 — committed to SWivid/F5-TTS by SWivid 2 years ago
- 0.5.0 fix grad_accum bug from 0.4.0, #715 #728 — committed to spygaurad/F5-TTS by SWivid 2 years ago
- 0.5.0 fix grad_accum bug from 0.4.0, #715 #728 — committed to deybith/F5-TTS by SWivid 2 years ago
@wangaocheng @P-uck @kdcyberdude @sarpba @campar @hcsolakoglu Hey guys, thanks for reporting the bug and all efforts.
Thought latest version (0.5.0) fixed this bug, sorry for the inconvenience caused !
Thank you! It’s ok now!
Same problem here too. Finetune hang on step one. (2xRTX3090+NVlink). The gpus power usage 150/370 Watt. Maybe stuck in loop forever.
@ZhikangNiu
Hi, i think the problem is that on Multi GPUs , the finetune process freezes , I have the same problem in " 715" as well,