交错式语音语言模型在文本中的潜在运作

摘要

语音语言模型（SLM）已被广泛研究，其常见范式是整合文本数据和预训练文本语言模型。一种主流方法是语音文本交错，即模型在包含语音和文本令牌的序列上进行训练，旨在提升甚至仅依赖语音的能力。然而，这两种模态在模型隐空间中的交互方式仍不明确。本研究通过logit透镜的视角，对来自不同模型族和规模的交错语音文本语言模型进行分析，以提供相关见解。我们发现，这些模型经历了一个隐式转录阶段——在此阶段中，口语词汇对应的文本令牌可在中间层被解码，尽管模型并未接受语音识别训练。在多达77%的数据中，该词汇的转录结果会出现在前几位候选词中。在此阶段之后，模型在文本空间中预测下一个单词，然后再转换回语音域。最后，我们分析了交错数据和文本语言模型初始化在引发这一行为中的作用，并探讨了该行为与口语知识能力的相关性。我们的分析揭示了语音与文本模态之间关系的内部机制，可能为SLM优化提供指导。

English

Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77\% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.