TESS：文本對文本自條件簡單擴散

摘要

擴散模型已成為一種強大的生成範式，能在具有連續值輸入的各個領域中取得優異表現。儘管完全非自回歸文本生成具有潛力，但將擴散模型應用於自然語言仍然具有挑戰性，因為其離散性質。在這項工作中，我們提出了一種名為Text-to-text Self-conditioned Simplex Diffusion (TESS) 的文本擴散模型，它是完全非自回歸的，採用一種新形式的自條件設定，並在對數函數空間中應用擴散過程，而非典型的學習嵌入空間。通過對自然語言理解和生成任務進行廣泛實驗，包括摘要、文本簡化、釋義生成和問題生成，我們展示了TESS優於最先進的非自回歸模型，並且與預訓練的自回歸序列到序列模型具有競爭力。

English

Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various domains with continuous-valued inputs. Despite the promises of fully non-autoregressive text generation, applying diffusion models to natural language remains challenging due to its discrete nature. In this work, we propose Text-to-text Self-conditioned Simplex Diffusion (TESS), a text diffusion model that is fully non-autoregressive, employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the typical learned embedding space. Through extensive experiments on natural language understanding and generation tasks including summarization, text simplification, paraphrase generation, and question generation, we demonstrate that TESS outperforms state-of-the-art non-autoregressive models and is competitive with pretrained autoregressive sequence-to-sequence models.

TESS：文本對文本自條件簡單擴散

TESS: Text-to-Text Self-Conditioned Simplex Diffusion

摘要

Support