規劃者：通過潛在語言擴散生成多樣化段落模型

摘要

基於自回歸模型的文本有時會生成重複且低質量的輸出，因為在生成步驟中錯誤會累積。這個問題通常被歸因於曝光偏差 - 模型在訓練時與推斷時的差異。去噪擴散模型提供了一種替代方法，其中模型可以重新訪問並修訂其輸出。然而，它們可能在計算上昂貴，先前在文本上的努力導致的模型生成的輸出比自回歸模型產生的輸出不太流暢，特別是對於較長的文本和段落。在本文中，我們提出了PLANNER，一種結合潛在語義擴散和自回歸生成的模型，以在段落上進行全局控制的方式生成流暢的文本。該模型通過將自回歸的“解碼”模塊與使用潛在擴散以粗到細的方式生成語義段落嵌入的“規劃”模塊相結合來實現這一目標。所提出的方法在各種條件生成任務上進行了評估，並在語義生成、文本補全和摘要方面的結果表明了其在以高效方式生成高質量長文本方面的有效性。

English

Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to exposure bias - the difference between how a model is trained, and how it is used during inference. Denoising diffusion models provide an alternative approach in which a model can revisit and revise its output. However, they can be computationally expensive and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module that uses latent diffusion to generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed method is evaluated on various conditional generation tasks, and results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text in an efficient manner.

規劃者：通過潛在語言擴散生成多樣化段落模型

PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model

摘要

Support