TTS: [Bug] Fine tuned XTTS v2 produces strange sounds for short text
Describe the bug
I have fine tuned XTTS v2 model on my own data containing both long and short audios (with the following histogram showing duration in seconds on x-axis. Labels ‘old’ and ‘new’ represent 2 datasets with long and short audios respectively.)
But the model produces strange sounds in case of 1-2 words text, like the following 2 examples for text='hola':
https://github.com/coqui-ai/TTS/assets/59258087/9e734e4b-3954-4adf-9919-7af42c8a28ad
https://github.com/coqui-ai/TTS/assets/59258087/f2f4b964-e1cd-4986-9f4c-d082a0a53d10
It seems like the model tries to produce at least 3 seconds audio even if the text is very short. And thus it adds some meaningless sounds to the sound of the original word in text.
@erogol Is there any way to avoid this behavior? or any parameter (may be in model args) to control this?
There are gpt_start_audio_token and gpt_stop_audio_token parameters in TTS.tts.models.xtts.XttsArgs() class but i am not sure what is the impact of these parameters?
To Reproduce
N/A
Expected behavior
Should produce short audio for short text.
Logs
No response
Environment
{
"CUDA": {
"GPU": [
"NVIDIA A30",
"NVIDIA A30",
"NVIDIA A30",
"NVIDIA A30"
],
"available": true,
"version": "12.1"
},
"Packages": {
"PyTorch_debug": false,
"PyTorch_version": "2.1.0+cu121",
"TTS": "0.22.0",
"numpy": "1.23.0"
},
"System": {
"OS": "Linux",
"architecture": [
"64bit",
"ELF"
],
"processor": "x86_64",
"python": "3.10.12",
"version": "#64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023"
}
}
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 6
- Comments: 24
Same issue. The pretrained Xtts v2 generate extra speech randomly.
Same issue. Pre-trained XTTSv2 produces extra speech after the intended “text”, 10-20% of the time