Customize-It-3D：使用主題特定知識先驗從單張圖像創建高質量3D模型

Customize-It-3D: High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Prior

December 15, 2023

作者: Nan Huang, Ting Zhang, Yuhui Yuan, Dong Chen, Shanghang Zhang

cs.AI

摘要

本文提出了一種新穎的兩階段方法，充分利用參考圖像提供的信息，建立一個定制的知識先驗，用於從圖像生成3D。傳統方法主要依賴於一般性擴散先驗，難以與參考圖像產生一致的結果，我們提出了一種主題特定和多模態擴散模型。該模型不僅通過考慮用於改善幾何形狀的著色模式來幫助 NeRF 優化，還從粗糙結果中增強紋理以實現卓越的細化。這兩個方面有助於將3D內容與主題忠實地對齊。大量實驗展示了我們的方法「定制化3D」的優越性，明顯優於先前的作品。它能夠生成具有出色視覺質量的忠實360度重建，非常適用於各種應用，包括從文本生成3D。

English

In this paper, we present a novel two-stage approach that fully utilizes the information provided by the reference image to establish a customized knowledge prior for image-to-3D generation. While previous approaches primarily rely on a general diffusion prior, which struggles to yield consistent results with the reference image, we propose a subject-specific and multi-modal diffusion model. This model not only aids NeRF optimization by considering the shading mode for improved geometry but also enhances texture from the coarse results to achieve superior refinement. Both aspects contribute to faithfully aligning the 3D content with the subject. Extensive experiments showcase the superiority of our method, Customize-It-3D, outperforming previous works by a substantial margin. It produces faithful 360-degree reconstructions with impressive visual quality, making it well-suited for various applications, including text-to-3D creation.