speechbrain: Training model with DDP couldn't accelerate
I used DDP to run Common and Vox recipes with 4-8 Gpus, but instead of speeding up, it took longer to train one epoch. My training environment is not NFS. Here is the script I ran
CUDA_VISIBLE_DEVICES=0,1,2,3 OMP_NUM_THREADS=40 python -m torch.distributed.launch --nproc_per_node=4 \
train.py hparams/train_ecapa_tdnn.yaml \
--data_folder=data/common_voice_kpd \
--batch_size=4 --seed=2000 \
--num_workers=20 \
--distributed_launch \
--distributed_backend='nccl'
Enlarging num_works mentioned in this issue In issue https://github.com/speechbrain/speechbrain/issues/784#issuecomment-847720463 does not solve my problem.
Is there something wrong? Thanks in advance!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 20
In the past we noticed that the ECAPA-TDNN model is slow when using multiple GPUs with DDP. As far as I remember, one user found out that this was related to the residual block used in the recipe (there was a discussion about that). There might be the need for a fix there. If someone from the community would like to help with that is welcome! The team has currently issues in supporting the speaker verification part because the students working on that left.
On Fri, 5 Nov 2021 at 12:38, lhanzl @.***> wrote: