Music-Source-Separation-Training: Accelerate version not working properly
I tried training a model using:
accelerate launch train_accelerate.py
I get this output:
/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 6
More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in --num_processes=1.
--num_machines was set to a value of 1
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
It continues to train after the warning, but the loss value is always ‘nan’ and the validation results in 0.0dB for all stems.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 26 (11 by maintainers)
In case you are using audiomentations 0.24.0 for data augmentation and you are observing bottleneck issues: I have improved the speed in audiomentations 0.27.0, 0.31.0, 0.34.1, 0.36.0, 0.36.1 and 0.37.0 (see changelog). Upgrading may help a little bit.
Looks like the problem was augmentations after all. Particularly the pitch and distortion. I didn’t realize I had these turned on. Now I see ~20% improvement when running with accelerate and there are no more pauses. I’m also able to get my data workers down to 2 with no problem. Thanks for the help!
If it’s pedalboard’s distortion that was slow, I would recommend to fully remove that augmentation as it’s also creating huge gain changes while audiomentations has better alternative like tanh that is gain-balanced, sounding more musical and faster. Most useful pedalboard augmentation is the reverb, for anything else I would go with audiomentations first.