AURORA-LM：用于连续潜在扩散语言建模的自编码统一表示

摘要

在生成建模中，语言仍然是一个特例：虽然图像、视频和音频日益在连续潜在空间中进行建模，但文本生成仍主要依赖离散词元。现有的连续语言模型要么继承了并非为联合生成与解码而设计的嵌入空间，要么压缩自编码的潜表示以简化扩散过程，从而牺牲了词元级保真度。我们没有为了适配生成模型而简化表示，而是保留了一个高容量、可解码的文本潜表示，并设计扩散模型直接学习其分布。我们提出了AURORA-LM，一种连续潜在空间扩散语言模型，它将可解码文本表示的构建与其分布的建模分离开来。基于查询的编码器-解码器将文本组织为高容量、前缀对齐的潜序列，而块因果扩散Transformer通过流匹配学习其分布，从左到右生成块，同时并行去噪每个块内的位置。由于这种潜表示对扩散建模更具挑战性，AURORA-LM仅限制噪声输入通路，同时保留完整的干净潜表示预测目标，从而在不降低解码器侧容量的情况下容纳全宽潜表示。我们还根据潜表示宽度校准噪声水平分布，并引入自轨迹一致性来弥合独立采样的训练噪声与推理时迭代去噪之间的差距。在OpenWebText自由生成和XSum摘要任务上，AURORA-LM在已评估的连续和基于扩散的语言模型中取得了最强性能。将模型扩展到10亿参数、总计算量约1500 EFLOPs可进一步提升性能，在对齐的评估协议下超越了已公开的更大规模潜扩散语言模型。所有实验均在昇腾NPU上进行。

English

Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.