Coin3D：使用代理引导调节的方式进行可控和交互式3D资产生成

Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning

May 13, 2024

作者: Wenqi Dong, Bangbang Yang, Lin Ma, Xiao Liu, Liyuan Cui, Hujun Bao, Yuewen Ma, Zhaopeng Cui

cs.AI

摘要

作为人类，我们渴望创造既具有自由意志又易于控制的媒体内容。由于生成技术的显著发展，我们现在可以轻松利用2D扩散方法合成由原始草图或指定人体姿势控制的图像，甚至可以逐步编辑/重建局部区域并进行遮罩修复。然而，在3D建模任务中类似的工作流程仍然不可用，这是由于3D生成中缺乏可控性和效率。在本文中，我们提出了一种新颖的可控交互式3D资产建模框架，命名为Coin3D。Coin3D允许用户使用由基本形状组装而成的粗略几何代理来控制3D生成，并引入交互式生成工作流程，支持无缝的局部部件编辑，同时在几秒钟内提供响应迅速的3D对象预览。为此，我们开发了几种技术，包括应用体积粗略形状控制于扩散模型的3D适配器，用于精确部件编辑的代理边界编辑策略，支持响应式预览的逐步体积缓存，以及用于确保一致网格重建的体积-SDS。对各种形状代理进行的交互式生成和编辑的大量实验表明，我们的方法在3D资产生成任务中实现了卓越的可控性和灵活性。

English

As humans, we aspire to create media content that is both freely willed and readily controlled. Thanks to the prominent development of generative techniques, we now can easily utilize 2D diffusion methods to synthesize images controlled by raw sketch or designated human poses, and even progressively edit/regenerate local regions with masked inpainting. However, similar workflows in 3D modeling tasks are still unavailable due to the lack of controllability and efficiency in 3D generation. In this paper, we present a novel controllable and interactive 3D assets modeling framework, named Coin3D. Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes, and introduces an interactive generation workflow to support seamless local part editing while delivering responsive 3D object previewing within a few seconds. To this end, we develop several techniques, including the 3D adapter that applies volumetric coarse shape control to the diffusion model, proxy-bounded editing strategy for precise part editing, progressive volume cache to support responsive preview, and volume-SDS to ensure consistent mesh reconstruction. Extensive experiments of interactive generation and editing on diverse shape proxies demonstrate that our method achieves superior controllability and flexibility in the 3D assets generation task.