Anchored Monotonic Alignment and Representation Substitution for Rare Spontaneous Behaviors in Spontaneous Speech Synthesis

Abstract

Spontaneous behaviors in speech present significant challenges for speech synthesis. Current research in this area has not sufficiently addressed these behaviors, and most studies rely on specially recorded data. In contrast, real-world data is generally more reflective of the natural, spontaneous speaking styles of daily life and exhibits a wider variety of spontaneous behaviors. However, this type of data is often not of studio-quality, and the distribution of different spontaneous behaviors is highly imbalanced. In this study, we investigate spontaneous speech synthesis using real-world data within the VITS2 framework. To handle rare spontaneous behaviors, we introduce two methods: anchored monotonic alignment and spontaneous hidden representation substitution. Experimental results show that these methods improve model alignment, resolve difficulties in synthesizing rare spontaneous behaviors, and significantly enhance the naturalness of synthesized speech.