文本扩散器：将扩散模型作为文本绘制工具

TextDiffuser: Diffusion Models as Text Painters

May 18, 2023

作者: Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei

cs.AI

摘要

扩散模型因其出色的生成能力而受到越来越多的关注，但目前在呈现准确和连贯的文本方面仍存在困难。为解决这一问题，我们引入了TextDiffuser，专注于生成具有视觉吸引力且与背景连贯的文本图像。TextDiffuser包括两个阶段：首先，一个Transformer模型生成从文本提示中提取的关键词的布局，然后扩散模型生成以文本提示和生成的布局为条件的图像。此外，我们贡献了第一个带有OCR注释的大规模文本图像数据集MARIO-10M，包含1000万个图像文本对，具有文本识别、检测和字符级分割注释。我们进一步收集了MARIO-Eval基准数据集，作为评估文本呈现质量的综合工具。通过实验和用户研究，我们展示了TextDiffuser具有灵活性和可控性，能够仅使用文本提示或结合文本模板图像创建高质量的文本图像，并进行文本修复以重建带有文本的不完整图像。代码、模型和数据集将在https://aka.ms/textdiffuser上提供。

English

Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. The code, model, and dataset will be available at https://aka.ms/textdiffuser.