FlexiSLM：一种动态可控帧率的语音语言模型

摘要

口语语言模型（SLMs）将大型语言模型（LLMs）扩展至语音输入与输出。现有SLMs以固定帧率（如25或12.5 Hz）表示语音，忽略了语音的时变信息密度，且在推理时无法灵活地在质量与速度之间进行权衡。最近的音频分词器研究提出了动态帧率语音编码，该技术利用这种非均匀性，实现了两个新能力：极低的平均帧率和帧率可控性。然而，该技术尚未应用于SLMs。我们提出灵活口语语言模型（FlexiSLM），这是首个在语音输入和输出上均支持动态且可控帧率的SLM。采用动态帧率表示后，FlexiSLM在其高质量工作点上优于包含Qwen2.5-Omni和Kimi-Audio在内的固定帧率7B模型。我们进一步验证了FlexiSLM可精确调控至4.0 Hz；在6.25 Hz时，其推理时间相比12.5 Hz大约减半，同时保持较强的语音到语音质量。音频样本可在 https://flexislm.github.io 获取。

English

Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .