Coin3D：使用代理引導條件控制和互動式生成3D資產

Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning

May 13, 2024

作者: Wenqi Dong, Bangbang Yang, Lin Ma, Xiao Liu, Liyuan Cui, Hujun Bao, Yuewen Ma, Zhaopeng Cui

cs.AI

摘要

作為人類，我們渴望創建既自由意志又容易控制的媒體內容。由於生成技術的顯著發展，我們現在可以輕鬆利用2D擴散方法來合成由原始草圖或指定人體姿勢控制的圖像，甚至可以逐步編輯/重建局部區域進行遮罩修補。然而，在3D建模任務中類似的工作流程仍然不可用，這是由於3D生成中缺乏可控性和效率。在本文中，我們提出了一個新穎的可控且互動的3D資產建模框架，名為Coin3D。Coin3D允許用戶使用由基本形狀組合而成的粗略幾何代理來控制3D生成，並引入互動生成工作流程，以支持無縫的局部部分編輯，同時在幾秒內提供響應式的3D物體預覽。為此，我們開發了幾種技術，包括將體積粗略形狀控制應用於擴散模型的3D適配器、用於精確部分編輯的代理邊界編輯策略、用於支持響應式預覽的逐步體積緩存，以及用於確保一致網格重建的體積-SDS。在各種形狀代理上的互動生成和編輯的大量實驗表明，我們的方法在3D資產生成任務中實現了卓越的可控性和靈活性。

English

As humans, we aspire to create media content that is both freely willed and readily controlled. Thanks to the prominent development of generative techniques, we now can easily utilize 2D diffusion methods to synthesize images controlled by raw sketch or designated human poses, and even progressively edit/regenerate local regions with masked inpainting. However, similar workflows in 3D modeling tasks are still unavailable due to the lack of controllability and efficiency in 3D generation. In this paper, we present a novel controllable and interactive 3D assets modeling framework, named Coin3D. Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes, and introduces an interactive generation workflow to support seamless local part editing while delivering responsive 3D object previewing within a few seconds. To this end, we develop several techniques, including the 3D adapter that applies volumetric coarse shape control to the diffusion model, proxy-bounded editing strategy for precise part editing, progressive volume cache to support responsive preview, and volume-SDS to ensure consistent mesh reconstruction. Extensive experiments of interactive generation and editing on diverse shape proxies demonstrate that our method achieves superior controllability and flexibility in the 3D assets generation task.