學習連續的3D詞彙以進行文本到圖像生成

Learning Continuous 3D Words for Text-to-Image Generation

February 13, 2024

作者: Ta-Ying Cheng, Matheus Gadelha, Thibault Groueix, Matthew Fisher, Radomir Mech, Andrew Markham, Niki Trigoni

cs.AI

摘要

目前對擴散模型（例如透過文本或控制網）進行影像生成的控制不足以識別抽象的連續屬性，如光線方向或非剛性形狀變化。本文提出一種方法，讓文本轉圖像模型的使用者能夠對圖像中的多個屬性進行精細控制。我們通過設計特殊的輸入標記集，可以連續地轉換這些標記集，我們稱之為連續3D詞。這些屬性可以例如被表示為滑塊，並與文本提示一起應用，以實現對影像生成的精細控制。我們展示，只需一個網格和一個渲染引擎，我們的方法可以被採用，以提供對幾個3D感知屬性的連續用戶控制，包括白天光線照射、鳥翼方向、遠近變焦效果和物體姿勢。我們的方法能夠同時條件影像創建，使用多個連續3D詞和文本描述，而不會給生成過程增加額外負擔。項目頁面：https://ttchengab.github.io/continuous_3d_words

English

Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words