TTS: [Bug] Fine tuned XTTS v2 produces strange sounds for short text

Describe the bug

I have fine tuned XTTS v2 model on my own data containing both long and short audios (with the following histogram showing duration in seconds on x-axis. Labels ‘old’ and ‘new’ represent 2 datasets with long and short audios respectively.)

data_es_mix_hist

But the model produces strange sounds in case of 1-2 words text, like the following 2 examples for text='hola':

https://github.com/coqui-ai/TTS/assets/59258087/9e734e4b-3954-4adf-9919-7af42c8a28ad

https://github.com/coqui-ai/TTS/assets/59258087/f2f4b964-e1cd-4986-9f4c-d082a0a53d10

It seems like the model tries to produce at least 3 seconds audio even if the text is very short. And thus it adds some meaningless sounds to the sound of the original word in text.

@erogol Is there any way to avoid this behavior? or any parameter (may be in model args) to control this? There are gpt_start_audio_token and gpt_stop_audio_token parameters in TTS.tts.models.xtts.XttsArgs() class but i am not sure what is the impact of these parameters?

To Reproduce

N/A

Expected behavior

Should produce short audio for short text.

Logs

No response

Environment

{
    "CUDA": {
        "GPU": [
            "NVIDIA A30",
            "NVIDIA A30",
            "NVIDIA A30",
            "NVIDIA A30"
        ],
        "available": true,
        "version": "12.1"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "2.1.0+cu121",
        "TTS": "0.22.0",
        "numpy": "1.23.0"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.10.12",
        "version": "#64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023"
    }
}

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 24

Most upvoted comments

Same issue. The pretrained Xtts v2 generate extra speech randomly.

Same issue. Pre-trained XTTSv2 produces extra speech after the intended “text”, 10-20% of the time