Build-A-Scene：用於擴散式圖像生成的互動式3D佈局控制

Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation

August 27, 2024

作者: Abdelrahman Eldesokey, Peter Wonka

cs.AI

摘要

我們提出了一種基於擴散的方法，用於具有互動式3D佈局控制的文本到圖像（T2I）生成。佈局控制已被廣泛研究，以減輕T2I擴散模型在理解對象的放置和與文本描述的關係方面的缺點。然而，現有的佈局控制方法僅限於2D佈局，需要用戶事先提供靜態佈局，並且無法在佈局更改時保留生成的圖像。這使得這些方法不適用於需要3D對象控制和迭代細化的應用，例如室內設計和複雜場景生成。為此，我們利用了最近在深度條件T2I模型方面的進展，提出了一種新穎的互動式3D佈局控制方法。我們將佈局控制中傳統的2D框替換為3D框。此外，我們將T2I任務改為多階段生成過程，在每個階段，用戶可以在3D中插入、更改和移動對象，同時保留早期階段的對象。我們通過我們提出的動態自注意（DSA）模塊和一致的3D對象平移策略來實現這一點。實驗表明，我們的方法可以基於3D佈局生成複雜場景，將對象生成成功率提高了2倍以上，超越了標準深度條件T2I方法。此外，與其他方法相比，在佈局更改時保留對象方面表現更好。項目頁面：https://abdo-eldesokey.github.io/build-a-scene/

English

We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control. Layout control has been widely studied to alleviate the shortcomings of T2I diffusion models in understanding objects' placement and relationships from text descriptions. Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout beforehand, and fail to preserve generated images under layout changes. This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g., interior design and complex scene generation. To this end, we leverage the recent advancements in depth-conditioned T2I models and propose a novel approach for interactive 3D layout control. We replace the traditional 2D boxes used in layout control with 3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation process, where at each stage, the user can insert, change, and move an object in 3D while preserving objects from earlier stages. We achieve this through our proposed Dynamic Self-Attention (DSA) module and the consistent 3D object translation strategy. Experiments show that our approach can generate complicated scenes based on 3D layouts, boosting the object generation success rate over the standard depth-conditioned T2I methods by 2x. Moreover, it outperforms other methods in comparison in preserving objects under layout changes. Project Page: https://abdo-eldesokey.github.io/build-a-scene/