StreamVoice：用於即時零樣本語音轉換的可串流上下文感知語言建模

StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion

January 19, 2024

作者: Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Zhuo Chen, Lei Xie, Yuping Wang, Yuxuan Wang

cs.AI

摘要

最近語言模型（LM）的進展展示了令人印象深刻的零樣本語音轉換（VC）性能。然而，現有基於LM的VC模型通常應用來自源語義到聲學特徵的離線轉換，需要完整的源語音，並限制了它們在實時應用中的部署。本文介紹了StreamVoice，一種新型的基於LM的流式模型，用於零樣本VC，實現了在給定任意說話者提示和源語音的情況下進行實時轉換。具體來說，為了實現流式處理能力，StreamVoice採用了一個完全因果上下文感知的LM，具有一個時間獨立的聲學預測器，同時在自回歸的每個時間步驟交替處理語義和聲學特徵，從而消除對完整源語音的依賴。為了應對流式處理中由於上下文不完整而可能導致的性能下降，我們通過兩種策略增強了LM的上下文感知性：1）教師引導的上下文預見，使用教師模型在訓練期間總結當前和未來的語義上下文，引導模型對缺失上下文的預測；2）語義遮罩策略，促進從前面受損的語義和聲學輸入進行聲學預測，增強上下文學習能力。值得注意的是，StreamVoice是第一個基於LM的流式零樣本VC模型，無需任何未來的前瞻。實驗結果表明，StreamVoice具有流式轉換能力，同時保持與非流式VC系統相當的零樣本性能。

English

Recent language model (LM) advancements have showcased impressive zero-shot voice conversion (VC) performance. However, existing LM-based VC models usually apply offline conversion from source semantics to acoustic features, demanding the complete source speech, and limiting their deployment to real-time applications. In this paper, we introduce StreamVoice, a novel streaming LM-based model for zero-shot VC, facilitating real-time conversion given arbitrary speaker prompts and source speech. Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporal-independent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech. To address the potential performance degradation from the incomplete context in streaming processing, we enhance the context-awareness of the LM through two strategies: 1) teacher-guided context foresight, using a teacher model to summarize the present and future semantic context during training to guide the model's forecasting for missing context; 2) semantic masking strategy, promoting acoustic prediction from preceding corrupted semantic and acoustic input, enhancing context-learning ability. Notably, StreamVoice is the first LM-based streaming zero-shot VC model without any future look-ahead. Experimental results demonstrate StreamVoice's streaming conversion capability while maintaining zero-shot performance comparable to non-streaming VC systems.