非語言發聲中的說話者身份：條件蒸餾與專家混合方法

摘要

隨著表達性文字轉語音（TTS）與語音轉換（VC）系統日益生成非語言發聲（NVVs）以提升自然度，可靠的說話者驗證（SV）變得至關重要，以便客觀評估口語與非口語片段中身分一致性之表現。然而，現行說話者驗證系統對非語言發聲的泛化能力不佳，且針對非語言發聲資料進行微調會導致語音表現出現災難性遺忘。本研究首次針對10種非語言發聲類型進行系統性探討，提出一套結合凍結Data2Vec自監督特徵與ECAPA-TDNN的架構，並透過具學習型領域感知路由的混合專家（MoE）模組加以強化。藉由預訓練教師模型在語音輸入上施加條件蒸餾損失，可保留語音對語音之準確性；而對比損失則能縮小語音與非語言發聲之間的領域差距。我們的方法將預訓練基準模型之語音與非語言發聲間的等錯誤率（EER）從38.93%降至22.66%，並透過蒸餾將語音等錯誤率從13.17%提升至9.24%。

English

As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.