Evaluating Robustness of Voice Conversion Systems under Multi-source Channel Interference

International Joint Conference on Neural Networks is the flagship conference of INNS. It brings together the brightest minds in neural networks and artificial intelligence, offering a platform for sharing ideas and fostering collaborations. IEEE/INNS IJCNN is a CCF-C conference.

Abstract: Voice Conversion (VC) technology holds significant potential for enhancing communication across varied application scenarios, such as voice chats, video conferencing, and VTuber live streaming. During the VC systems’ use, there is inevitable and complex channel interference, which becomes a key factor in downgrading the performance of VC systems. However, few studies comprehensively reveal the specific impact of these interferences on VC systems, which gradually become a significant gap between VC design and its landing application requirements. Toward this end, this paper proposes a comprehensive evaluation framework to systematically analyze the effects of multi-source channel interference on VC systems. We investigate multi-source channel interference across physical and digital domains, including noise, reverberation, device distortion, codec compression and transmission loss, and integrate all of them into our proposed evaluation framework. The framework assesses VC systems in terms of three complementary dimensions, i.e., the speaker timbre, speech semantics, and signal consistency, by introducing three core metrics: timbre similarity, semantic similarity, and acoustic similarity. Through extensive experiments on 44,000 samples from 109 speakers across six representative VC systems, we find that VC systems: 1) handle digital interference better than physical interference, 2) maintain timbre features well but struggle with semantics and sound quality, and 3) could achieve robust conversion through feature disentanglement.

View the full paper