交錯式語音語言模型在文本中的潛在運作

摘要

語音語言模型（SLMs）已被廣泛研究，常見的範式會結合文本數據與預訓練文本語言模型。其中一種主流方法是語音-文本交錯訓練，讓模型在包含語音與文本標記的序列上進行訓練，旨在提升甚至僅處理語音的能力。然而，這兩種模態在模型潛在空間中如何互動仍不清楚。本研究透過logit透鏡的視角，分析來自不同模型家族與規模的交錯式語音-文本語言模型，以提供相關洞見。我們發現，這些模型會經歷一個隱式轉錄階段——即便未經語音辨識訓練，中間層仍能解碼出對應語詞的文本標記。在高達77%的數據中，該詞語的轉錄結果會出現在前幾名候選詞之列。在此階段之後，模型會先在文本空間中預測下一個詞語，然後再轉換回語音領域。最後，我們分析了交錯訓練數據與基於文本語言模型初始化在引發此行為中的作用，並探討此行為與語音知識能力之間的關聯。我們的分析揭示了語音與文本模態之間內部機制的運作方式，有助於優化語音語言模型的設計。

English

Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.