ThemeStation：從少量範例生成主題感知的3D資產

ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars

March 22, 2024

作者: Zhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, Rynson W. H. Lau

cs.AI

摘要

現實世界的應用通常需要一個龐大的 3D 資產庫，這些資產具有一致的主題。儘管在從文本或圖像中創建一般 3D 內容方面取得了顯著進展，但根據輸入 3D 示例合成符合共享主題的定制 3D 資產仍然是一個開放且具有挑戰性的問題。在這項工作中，我們提出了一種新方法 ThemeStation，用於主題感知的 3D 到 3D 生成。ThemeStation 根據少量給定示例合成定制的 3D 資產，具有兩個目標：1) 一致性，用於生成與給定示例在主題上對齊的 3D 資產，以及 2) 多樣性，用於生成具有高度變化的 3D 資產。為此，我們設計了一個兩階段框架，首先繪製概念圖像，然後進行參考資訊引導的 3D 建模階段。我們提出了一種新的雙分數蒸餾（DSD）損失，以共同利用來自輸入示例和合成概念圖像的先驗知識。大量實驗和用戶研究證實，ThemeStation 在生成具有印象深刻質量的多樣主題感知 3D 模型方面勝過先前的工作。ThemeStation 還支持各種應用，如可控的 3D 到 3D 生成。

English

Real-world applications often require a large gallery of 3D assets that share a consistent theme. While remarkable advances have been made in general 3D content creation from text or image, synthesizing customized 3D assets following the shared theme of input 3D exemplars remains an open and challenging problem. In this work, we present ThemeStation, a novel approach for theme-aware 3D-to-3D generation. ThemeStation synthesizes customized 3D assets based on given few exemplars with two goals: 1) unity for generating 3D assets that thematically align with the given exemplars and 2) diversity for generating 3D assets with a high degree of variations. To this end, we design a two-stage framework that draws a concept image first, followed by a reference-informed 3D modeling stage. We propose a novel dual score distillation (DSD) loss to jointly leverage priors from both the input exemplars and the synthesized concept image. Extensive experiments and user studies confirm that ThemeStation surpasses prior works in producing diverse theme-aware 3D models with impressive quality. ThemeStation also enables various applications such as controllable 3D-to-3D generation.