ThemeStation：从少量示例生成主题感知的3D资产

ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars

March 22, 2024

作者: Zhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, Rynson W. H. Lau

cs.AI

摘要

现实世界的应用通常需要一个庞大的 3D 资产库，这些资产共享一致的主题。虽然在从文本或图像中创建一般 3D 内容方面取得了显著进展，但根据输入的 3D 样本合成符合共享主题的定制 3D 资产仍然是一个未解决且具有挑战性的问题。在这项工作中，我们提出了 ThemeStation，这是一种新颖的主题感知 3D 到 3D 生成方法。ThemeStation 根据给定的少量样本合成定制的 3D 资产，具有两个目标：1）生成与给定样本在主题上一致的 3D 资产的统一性，2）生成具有高度变化的 3D 资产的多样性。为此，我们设计了一个两阶段框架，首先绘制一个概念图像，然后是一个基于参考信息的 3D 建模阶段。我们提出了一种新颖的双分数蒸馏（DSD）损失，共同利用来自输入样本和合成概念图像的先验知识。大量实验和用户研究证实，ThemeStation 在生成多样化的主题感知 3D 模型方面超越了先前的工作，并具有令人印象深刻的质量。ThemeStation 还支持各种应用，如可控的 3D 到 3D 生成。

English

Real-world applications often require a large gallery of 3D assets that share a consistent theme. While remarkable advances have been made in general 3D content creation from text or image, synthesizing customized 3D assets following the shared theme of input 3D exemplars remains an open and challenging problem. In this work, we present ThemeStation, a novel approach for theme-aware 3D-to-3D generation. ThemeStation synthesizes customized 3D assets based on given few exemplars with two goals: 1) unity for generating 3D assets that thematically align with the given exemplars and 2) diversity for generating 3D assets with a high degree of variations. To this end, we design a two-stage framework that draws a concept image first, followed by a reference-informed 3D modeling stage. We propose a novel dual score distillation (DSD) loss to jointly leverage priors from both the input exemplars and the synthesized concept image. Extensive experiments and user studies confirm that ThemeStation surpasses prior works in producing diverse theme-aware 3D models with impressive quality. ThemeStation also enables various applications such as controllable 3D-to-3D generation.