词元时间连续扩散语言建模

摘要

本文提出令牌时间连续扩散模型（Token Time Continuous Diffusion, TTCD），这是一种新型扩散语言模型，其核心特性包括：（a）在连续空间中运行，将高斯噪声确定性映射为最终令牌画布，无需额外采样；（b）引入全新的逐令牌时间概念，部分令牌从噪声到令牌的转化速率快于其他令牌。连续空间建模使TTCD能够避免多令牌并行采样，而并行采样正是纯离散空间模型在高加速场景下精度损失的关键原因。逐令牌时间概念帮助TTCD更好地建模条件生成，允许置信度更高的令牌以更快速率推进，并在精炼过程中实现差异化的令牌间交互影响。在高加速条件下，TTCD性能优于离散模型。我们在OpenWebText数据集上训练了1.6亿参数的TTCD模型并对其进行自蒸馏；实验表明，在高加速条件下，我们的模型在无条件生成质量上与现有多个同等规模、同数据集训练并自蒸馏的模型相当，在条件生成任务中则表现更优。此外，我们在数独求解任务中也获得了类似的性能提升。

English

In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.