MatSpray：在三维几何体上融合二维材料世界知识

摘要

在游戏和电影行业中，手动建模材质参数与三维几何是一项耗时但至关重要的任务。尽管三维重建技术的最新进展已能实现对场景几何与外观的精确近似，但这些方法因缺乏精确的空间变化材质参数，在重光照场景中往往表现不佳。与此同时，基于二维图像的扩散模型在预测基于物理的渲染（PBR）属性（如漫反射率、粗糙度和金属度）方面展现出强大性能。然而，将这些二维材质贴图迁移至重建的三维几何体仍面临重大挑战。我们提出了一种融合学习式与投影式创新方法的框架，将二维材质数据融入三维几何体。该框架首先通过高斯泼溅技术重建场景几何，再利用扩散模型从输入图像生成漫反射率、粗糙度和金属度的二维贴图（任何能将图像或视频转换为PBR材质的现有扩散模型均可适用）。预测结果通过两种方式融入三维表征：一是优化基于图像的损失函数，二是借助高斯光线追踪直接将材质参数投影至高斯单元。为提升微观尺度精度与多视角一致性，我们进一步引入轻量级神经优化步骤（神经融合器），该模块以光线追踪生成的材质特征为输入，输出细节调整量。实验结果表明，所提方法在量化指标与视觉真实感方面均优于现有技术，能够从重建场景中生成更精确、可重光照且具有照片级真实感的渲染效果，显著提升了内容生产流程中资产创建工作流的真实性与效率。

English

Manual modeling of material parameters and 3D geometry is a time consuming yet essential task in the gaming and film industries. While recent advances in 3D reconstruction have enabled accurate approximations of scene geometry and appearance, these methods often fall short in relighting scenarios due to the lack of precise, spatially varying material parameters. At the same time, diffusion models operating on 2D images have shown strong performance in predicting physically based rendering (PBR) properties such as albedo, roughness, and metallicity. However, transferring these 2D material maps onto reconstructed 3D geometry remains a significant challenge. We propose a framework for fusing 2D material data into 3D geometry using a combination of novel learning-based and projection-based approaches. We begin by reconstructing scene geometry via Gaussian Splatting. From the input images, a diffusion model generates 2D maps for albedo, roughness, and metallic parameters. Any existing diffusion model that can convert images or videos to PBR materials can be applied. The predictions are further integrated into the 3D representation either by optimizing an image-based loss or by directly projecting the material parameters onto the Gaussians using Gaussian ray tracing. To enhance fine-scale accuracy and multi-view consistency, we further introduce a light-weight neural refinement step (Neural Merger), which takes ray-traced material features as input and produces detailed adjustments. Our results demonstrate that the proposed methods outperform existing techniques in both quantitative metrics and perceived visual realism. This enables more accurate, relightable, and photorealistic renderings from reconstructed scenes, significantly improving the realism and efficiency of asset creation workflows in content production pipelines.

MatSpray：在三维几何体上融合二维材料世界知识

MatSpray: Fusing 2D Material World Knowledge on 3D Geometry

摘要

Support