使用协同控制共同生成多视角一致的PBR纹理

Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control

October 9, 2024

作者: Shimon Vainer, Konstantin Kutsy, Dante De Nigris, Ciara Rowles, Slava Elizarov, Simon Donné

cs.AI

摘要

图像扩散模型中的多视图一致性仍然是一个挑战。即使在已知先验几何对应关系的文本到纹理问题中，许多方法也无法产生跨视图对齐的预测，需要使用非平凡的融合方法将结果合并到原始网格上。我们特别探讨了在基于物理的渲染（PBR）文本到纹理中的协作控制工作流中的这个问题。协作控制直接对PBR图像概率分布进行建模，包括法线凹凸贴图；据我们所知，这是唯一直接输出完整PBR堆栈的扩散模型。我们讨论了设计决策，使该模型实现多视图一致，并通过消融研究以及实际应用展示了我们方法的有效性。

English

Multi-view consistency remains a challenge for image diffusion models. Even within the Text-to-Texture problem, where perfect geometric correspondences are known a priori, many methods fail to yield aligned predictions across views, necessitating non-trivial fusion methods to incorporate the results onto the original mesh. We explore this issue for a Collaborative Control workflow specifically in PBR Text-to-Texture. Collaborative Control directly models PBR image probability distributions, including normal bump maps; to our knowledge, the only diffusion model to directly output full PBR stacks. We discuss the design decisions involved in making this model multi-view consistent, and demonstrate the effectiveness of our approach in ablation studies, as well as practical applications.