MaGRITTe：從圖像、俯視圖和文本實現的操控性和生成性3D建模

MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Text

March 30, 2024

作者: Takayuki Hara, Tatsuya Harada

cs.AI

摘要

從使用者指定的條件生成3D場景為減輕3D應用程式中的製作負擔提供了一個有前途的途徑。先前的研究需要大量努力來實現所需的場景，這是由於受限的控制條件。我們提出了一種方法，使用部分圖像、在俯視圖中表示的佈局信息和文本提示來控制和生成多模態條件下的3D場景。將這些條件結合起來生成3D場景涉及以下重要困難：(1) 創建大型數據集，(2) 反映多模態條件的交互作用，以及(3) 佈局條件的領域依賴性。我們將3D場景生成過程分解為從給定條件生成2D圖像和從2D圖像生成3D場景。通過微調預訓練的文本到圖像模型，使用部分圖像和佈局的小型人工數據集實現2D圖像生成，並通過基於佈局的深度估計和神經輻射場(NeRF)實現3D場景生成，從而避免創建大型數據集。使用360度圖像來共同表示空間信息，有助於考慮多模態條件的交互作用，並減少佈局控制的領域依賴性。實驗結果在質量和量化上顯示，所提出的方法可以根據多模態條件在各種領域生成3D場景，從室內到室外。

English

The generation of 3D scenes from user-specified conditions offers a promising avenue for alleviating the production burden in 3D applications. Previous studies required significant effort to realize the desired scene, owing to limited control conditions. We propose a method for controlling and generating 3D scenes under multimodal conditions using partial images, layout information represented in the top view, and text prompts. Combining these conditions to generate a 3D scene involves the following significant difficulties: (1) the creation of large datasets, (2) reflection on the interaction of multimodal conditions, and (3) domain dependence of the layout conditions. We decompose the process of 3D scene generation into 2D image generation from the given conditions and 3D scene generation from 2D images. 2D image generation is achieved by fine-tuning a pretrained text-to-image model with a small artificial dataset of partial images and layouts, and 3D scene generation is achieved by layout-conditioned depth estimation and neural radiance fields (NeRF), thereby avoiding the creation of large datasets. The use of a common representation of spatial information using 360-degree images allows for the consideration of multimodal condition interactions and reduces the domain dependence of the layout control. The experimental results qualitatively and quantitatively demonstrated that the proposed method can generate 3D scenes in diverse domains, from indoor to outdoor, according to multimodal conditions.