统一音频智能而不牺牲文本智能

摘要

音频智能涉及对音频和语音的理解、推理及生成。本文介绍Nemotron-Labs-Audex-30B-A3B（简称Audex），这是一个基于Nemotron-Cascade-2-30B-A3B（一款强大的纯文本MoE大语言模型）构建的统一音频-文本大语言模型。Audex采用简洁的统一设计，仅包含单个Transformer解码器：音频输入经过编码并投影到文本嵌入空间，而文本令牌与量化音频输出令牌在生成过程中被统一对待。这种架构实现了强大的音频-文本融合、无缝的多模态生成，并与标准大语言模型的训练和推理基础设施兼容。在训练方面，我们精心策划了包含1574亿音频令牌和3205亿文本令牌的音频-文本数据集，并对这些数据集进行多阶段监督训练，随后进行纯文本Cascade强化学习及多领域在线策略蒸馏。Audex在音频理解、语音识别与翻译、文本到语音、音频生成及语音到语音生成方面均达到最先进水平，同时其纯文本大语言模型主干所具备的推理、对齐、知识、长上下文及智能体能力得以几乎完全保留，仅出现极轻微或无回归。我们发布模型检查点以促进开放研究。

English

Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.