speechbrain: training on voxceleb1+2 is very slow?
Dear all: I noticed that when training on voxceleb1+2, it will take me up to 25 hours for single epoch. and even with ddp on 4 gpu cards, the training speed does not reduce at all. I guess the cpu is the bottleneck? anyone has the same phenomena? thank you.
7%|████████▎ | 16569/241547 [1:45:07<25:09:56, 2.48it/s, train_loss=13
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 35 (4 by maintainers)
On LIA clusters you can increase this value as our nfs is pretty terrible. (4-8) to be tested. It’s the number of threads used to load the data. Having 100% usage is pretty normal as well, but the training should not be that long.