MaterialFusion：通过材质扩散增强反渲染 先验

MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors

September 23, 2024

作者: Yehonathan Litman, Or Patashnik, Kangle Deng, Aviral Agrawal, Rushikesh Zawar, Fernando De la Torre, Shubham Tulsiani

cs.AI

摘要

最近的反渲染研究表明利用物体的多视图图像恢复形状、反照率和材质具有潜力。然而，由于从输入图像中解开反照率和材质属性的固有挑战，恢复的组件通常无法在新的光照条件下准确渲染。为了解决这一挑战，我们引入了MaterialFusion，这是一个增强的传统3D反渲染流程，融合了对纹理和材质属性的2D先验。我们提出了StableMaterial，这是一个2D扩散模型先验，用于优化多光照数据，从给定的输入外观中估计最可能的反照率和材质。该模型是在一个由约12K个艺术家设计的合成Blender对象组成的策划数据集BlenderVault中，通过反照率、材质和重照图像数据进行训练的。我们将这种扩散先验与反渲染框架相结合，其中我们使用得分蒸馏采样（SDS）来引导反照率和材质的优化，从而提高了与先前工作相比的重照性能。我们在4个合成和真实物体的数据集上验证了MaterialFusion在不同照明条件下的重照性能，显示我们的扩散辅助方法显著改善了在新的光照条件下重建物体的外观。我们打算公开发布我们的BlenderVault数据集，以支持这一领域的进一步研究。

English

Recent works in inverse rendering have shown promise in using multi-view images of an object to recover shape, albedo, and materials. However, the recovered components often fail to render accurately under new lighting conditions due to the intrinsic challenge of disentangling albedo and material properties from input images. To address this challenge, we introduce MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that incorporates a 2D prior on texture and material properties. We present StableMaterial, a 2D diffusion model prior that refines multi-lit data to estimate the most likely albedo and material from given input appearances. This model is trained on albedo, material, and relit image data derived from a curated dataset of approximately ~12K artist-designed synthetic Blender objects called BlenderVault. we incorporate this diffusion prior with an inverse rendering framework where we use score distillation sampling (SDS) to guide the optimization of the albedo and materials, improving relighting performance in comparison with previous work. We validate MaterialFusion's relighting performance on 4 datasets of synthetic and real objects under diverse illumination conditions, showing our diffusion-aided approach significantly improves the appearance of reconstructed objects under novel lighting conditions. We intend to publicly release our BlenderVault dataset to support further research in this field.