GuideFlow3D：优化导向的整流流用于外观迁移

摘要

利用外觀物體的不同表徵——如圖像或文本——將外觀轉移至三維資產上，因其在遊戲、增強現實及數字內容創作等行業的廣泛應用而受到關注。然而，當輸入與外觀物體之間的幾何結構存在顯著差異時，現有的尖端方法仍難以應對。一種直觀的策略是直接應用三維生成模型，但我們證明這最終無法產生令人滿意的結果。相反，我們提出了一種受通用指導啟發的原則性方法。基於預訓練的、以圖像或文本為條件的校正流模型，我們無需訓練的方法通過定期添加指導來與採樣過程互動。此指導可建模為可微分的損失函數，我們嘗試了兩種不同類型的指導，包括針對外觀的部分感知損失和自相似性。實驗表明，我們的方法成功將紋理和幾何細節轉移至輸入的三維資產上，在質量和數量上均超越了基準方法。我們還指出，由於傳統指標無法聚焦於局部細節且在缺乏真實數據的情況下比較不同輸入，它們並不適合評估此任務。因此，我們採用基於GPT的系統客觀地對輸出進行排名，以評估外觀轉移的質量，確保評估的穩健性和人性化，這一點在我們的用戶研究中得到了進一步證實。除了展示的場景外，我們的方法具有通用性，可擴展至不同類型的擴散模型和指導函數。

English

Transferring appearance to 3D assets using different representations of the appearance object - such as images or text - has garnered interest due to its wide range of applications in industries like gaming, augmented reality, and digital content creation. However, state-of-the-art methods still fail when the geometry between the input and appearance objects is significantly different. A straightforward approach is to directly apply a 3D generative model, but we show that this ultimately fails to produce appealing results. Instead, we propose a principled approach inspired by universal guidance. Given a pretrained rectified flow model conditioned on image or text, our training-free method interacts with the sampling process by periodically adding guidance. This guidance can be modeled as a differentiable loss function, and we experiment with two different types of guidance including part-aware losses for appearance and self-similarity. Our experiments show that our approach successfully transfers texture and geometric details to the input 3D asset, outperforming baselines both qualitatively and quantitatively. We also show that traditional metrics are not suitable for evaluating the task due to their inability of focusing on local details and comparing dissimilar inputs, in absence of ground truth data. We thus evaluate appearance transfer quality with a GPT-based system objectively ranking outputs, ensuring robust and human-like assessment, as further confirmed by our user study. Beyond showcased scenarios, our method is general and could be extended to different types of diffusion models and guidance functions.

GuideFlow3D：优化导向的整流流用于外观迁移

GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer

摘要

Support