詞元時間連續擴散用於語言建模

摘要

本文提出令牌时间连续扩散（TTCD），一种新型扩散语言模型，其核心特性包括：（a）在连续空间中运行，将高斯噪声确定性地映射至最终令牌画布，无需额外采样；以及（b）引入全新的逐令牌时间概念，使得部分令牌从噪声到令牌的推进速率快于其他令牌。连续空间建模有助于TTCD避免并行采样多个令牌——这是纯离散空间迭代模型在高加速比下出现不准确性的关键来源。逐令牌时间概念使TTCD能更好地建模条件生成，允许确定性更高的令牌以更快速率推进，并在精炼过程中实现差异化的令牌间相互影响。在高加速比下，TTCD表现优于离散模型。我们在OpenWebText上训练了1.6亿参数的TTCD模型，并对其进行自蒸馏；结果发现，在高加速比下，我们在无条件生成质量上与若干同规模、同数据训练并自蒸馏的现有模型相当，且在条件生成中表现更优。在数独求解任务中，我们也取得了类似的性能提升。

English

In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.