城市建築師：具有佈局先驗的可導向式3D城市場景生成

Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

April 10, 2024

作者: Fan Lu, Kwan-Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, Changjun Jiang

cs.AI

摘要

透過大規模的文本到圖像擴散模型，文本到3D生成已取得顯著成功。然而，目前尚無法將這種方法融入城市尺度。城市場景以眾多元素、複雜的排列關係和龐大的尺度為特徵，對於模型優化的有效性構成了難以理解的模糊文本描述的障礙。在本研究中，我們通過將組合式3D佈局表示引入文本到3D範式，克服了這些限制，作為額外的先驗。它包括一組具有簡單幾何結構和明確排列關係的語義基元，補充了文本描述，實現了可操控的生成。基於此，我們提出了兩項修改：(1)我們引入佈局引導變分分數蒸餾，以解決模型優化不足的問題。它通過3D佈局的幾何和語義約束來條件化分數蒸餾採樣過程。(2)為應對城市場景的無限性，我們使用可擴展的哈希網格結構來表示3D場景，逐步適應城市場景不斷增長的尺度。大量實驗證實了我們的框架將文本到3D生成擴展至覆蓋超過1000米行駛距離的大規模城市場景的能力。我們還展示了各種場景編輯演示，展示了可操控的城市場景生成的威力。網站：https://urbanarchitect.github.io。

English

Text-to-3D generation has achieved remarkable success via large-scale text-to-image diffusion models. Nevertheless, there is no paradigm for scaling up the methodology to urban scale. Urban scenes, characterized by numerous elements, intricate arrangement relationships, and vast scale, present a formidable barrier to the interpretability of ambiguous textual descriptions for effective model optimization. In this work, we surmount the limitations by introducing a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships, complementing textual descriptions and enabling steerable generation. Upon this, we propose two modifications -- (1) We introduce Layout-Guided Variational Score Distillation to address model optimization inadequacies. It conditions the score distillation sampling process with geometric and semantic constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes, we represent 3D scene with a Scalable Hash Grid structure, incrementally adapting to the growing scale of urban scenes. Extensive experiments substantiate the capability of our framework to scale text-to-3D generation to large-scale urban scenes that cover over 1000m driving distance for the first time. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation. Website: https://urbanarchitect.github.io.