使用協同控制共同生成多視角一致的PBR紋理

Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control

October 9, 2024

作者: Shimon Vainer, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Slava Elizarov, Simon Donné

cs.AI

摘要

在影像擴散模型中，多視角一致性仍然是一個挑戰。即使在已知先驗下的文本轉紋理問題中，許多方法也無法產生跨視角對齊的預測，需要使用非平凡的融合方法將結果整合到原始網格上。我們專注於 PBR 文本轉紋理中的協同控制工作流程中的這個問題。協同控制直接建模 PBR 影像概率分佈，包括法線凸起貼圖；據我們所知，這是唯一直接輸出完整 PBR 堆疊的擴散模型。我們討論了設計這個模型以實現多視角一致性所涉及的決策，並通過消融研究和實際應用展示了我們方法的有效性。

English

Multi-view consistency remains a challenge for image diffusion models. Even within the Text-to-Texture problem, where perfect geometric correspondences are known a priori, many methods fail to yield aligned predictions across views, necessitating non-trivial fusion methods to incorporate the results onto the original mesh. We explore this issue for a Collaborative Control workflow specifically in PBR Text-to-Texture. Collaborative Control directly models PBR image probability distributions, including normal bump maps; to our knowledge, the only diffusion model to directly output full PBR stacks. We discuss the design decisions involved in making this model multi-view consistent, and demonstrate the effectiveness of our approach in ablation studies, as well as practical applications.