NeMo: problem with fine-tuning th e pre-trained `SpeakerNet` model for speaker recognition
Hi!
As an experiment, I was fine-tuning the pre-trained SpeakerNet
model with the an4
dataset plus my own voice recordings. I used the example script speaker_reco_finetune.py
and changed the configuration file according to the documentation and the colab notebook tutorial. But it appears something is always wrong with setting up the optimizer:
Traceback (most recent call last):
File "/Users/xujinghua/speaker-recognition-nemo/scripts/experimental/fine_tune_an4.py", line 119, in <module>
main()
File "/Users/xujinghua/speaker-recognition-nemo/scripts/experimental/fine_tune_an4.py", line 108, in main
speaker_model.setup_optimization(speaker_model._cfg.optim)
File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/nemo/core/classes/modelPT.py", line 603, in setup_optimization
optimizer=self._optimizer, scheduler_config=scheduler_config, train_dataloader=self._train_dl
File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/nemo/core/optim/lr_scheduler.py", line 539, in prepare_lr_scheduler
max_epochs, accumulate_grad_batches, num_workers, num_samples, batch_size, drop_last,
File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/nemo/core/optim/lr_scheduler.py", line 587, in compute_max_steps
sampler_num_samples = math.ceil(num_samples / num_workers)
ZeroDivisionError: division by zero
I tried a few ways to debug but nothing helped I have no clue what is going on wrong. Has anyone got an idea or run into such problems before? Help would be super appreciated!
Cheers, JX
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15
Hi @JINHXu … To finetune on cpu, you have to make sure to set accelerator to
null
in config, along with gpus=0. This should work