我剛剛說了什麼？全雙工語音模型的自我聆聽

摘要

全雙工口語語言模型能夠同時聆聽與說話，使其得以處理人類對話中的打斷與回饋訊號。然而，文字生成、語音合成與音訊播放係以非同步方式進行。因此，模型自認為已經說出的內容，可能與實際播放給使用者的內容不一致。我們將在從中斷中恢復時仍能掌握模型實際已播放語音的問題，稱為錨定中斷。為解決此問題，我們提出 Self-Listening（自聆聽），一種全雙工建模方法，將使用者語音、模型文字與模型已播放語音交錯處理。透過將已實際播放的語音輸出作為輸入串流回饋給模型，自聆聽使中斷恢復奠基於使用者實際聽到的內容。我們進一步提出 AnchorSpeech，一個具有同質訓練與測試分割的資料集，用於追蹤結構化有序回應中哪些項目實際上已被說出。AnchorSpeech-test 評估模型是否能以中斷前最後完成的項目為依據，做出一致的回應。實驗顯示，與全雙工基線相比，配備自聆聽機制的模型能達到更佳的錨定效能。

English

Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.