TESS：文本到文本自条件简单扩散

摘要

扩散模型已经成为一种强大的生成范式，在具有连续值输入的各个领域取得了出色的性能。尽管完全非自回归文本生成具有潜力，但由于其离散性质，将扩散模型应用于自然语言仍然具有挑战性。在这项工作中，我们提出了一种名为文本自条件简单形式扩散（TESS）的文本扩散模型，它是完全非自回归的，采用一种新形式的自条件，并在对数几率单纯形空间上应用扩散过程，而不是典型的学习嵌入空间。通过对自然语言理解和生成任务进行广泛实验，包括摘要、文本简化、释义生成和问题生成，我们证明了TESS优于最先进的非自回归模型，并且与预训练的自回归序列到序列模型具有竞争力。

English

Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various domains with continuous-valued inputs. Despite the promises of fully non-autoregressive text generation, applying diffusion models to natural language remains challenging due to its discrete nature. In this work, we propose Text-to-text Self-conditioned Simplex Diffusion (TESS), a text diffusion model that is fully non-autoregressive, employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the typical learned embedding space. Through extensive experiments on natural language understanding and generation tasks including summarization, text simplification, paraphrase generation, and question generation, we demonstrate that TESS outperforms state-of-the-art non-autoregressive models and is competitive with pretrained autoregressive sequence-to-sequence models.

TESS：文本到文本自条件简单扩散

TESS: Text-to-Text Self-Conditioned Simplex Diffusion

摘要

Support