使用多光照合成的扩散方法进行辐射场照明重新照明

摘要

对于多视角数据，重新照明辐射场存在严重的欠约束问题，因为大多数情况下这些数据是在单一照明条件下捕获的；对于包含多个物体的完整场景尤其困难。我们提出了一种方法，通过利用从2D图像扩散模型中提取的先验知识，使用这种单一照明数据来创建可重新照明的辐射场。我们首先在一个以光照方向为条件的多照明数据集上对2D扩散模型进行微调，从而能够将单一照明捕获转换为一个逼真但可能不一致的多照明数据集，直接定义光照方向。我们利用这个增强数据来创建由3D高斯斑点表示的可重新照明的辐射场。为了实现对低频照明方向的直接控制，我们使用一个以光照方向为参数的多层感知器来表示外观。为了强化多视角一致性并克服不准确性，我们优化了每个图像的辅助特征向量。我们展示了在单一照明条件下对合成和真实多视角数据的结果，表明我们的方法成功利用了2D扩散模型的先验知识，实现了对完整场景进行逼真的3D重新照明。项目网站：https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/

English

Relighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic -- but possibly inconsistent -- multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. Project site https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/

使用多光照合成的扩散方法进行辐射场照明重新照明

A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

摘要

Support