动态网格感知辐射场

Dynamic Mesh-Aware Radiance Fields

September 8, 2023

作者: Yi-Ling Qiao, Alexander Gao, Yiran Xu, Yue Feng, Jia-Bin Huang, Ming C. Lin

cs.AI

摘要

在将多边形网格资产嵌入逼真的神经辐射场（NeRF）体积中，以便能够以与NeRF物理一致的方式渲染它们并模拟它们的动态，这是从将NeRF整合到传统图形管线的系统角度来看尚未充分探讨的问题。本文设计了在渲染和模拟过程中网格和NeRF之间的双向耦合。我们首先回顾了网格和NeRF的光传输方程，然后将它们提炼成一种高效算法，用于沿着投射射线更新辐射和吞吐量，可以进行任意次反射。为了解决路径追踪器假定的线性颜色空间与标准NeRF使用的sRGB颜色空间之间的差异，我们使用高动态范围（HDR）图像对NeRF进行训练。我们还提出了一种估计光源并在NeRF上投射阴影的策略。最后，我们考虑了如何将混合表面-体积形式与支持布料、刚体和软体的高性能物理模拟器有效集成。完整的渲染和模拟系统可以在GPU上以交互速率运行。我们展示了混合系统方法在网格插入方面胜过其他替代方案，因为它允许从体积NeRF介质到表面进行逼真的光传输，这影响了反射/折射表面的外观和由动态场景信息驱动的漫反射表面的照明。

English

Embedding polygonal mesh assets within photorealistic Neural Radience Fields (NeRF) volumes, such that they can be rendered and their dynamics simulated in a physically consistent manner with the NeRF, is under-explored from the system perspective of integrating NeRF into the traditional graphics pipeline. This paper designs a two-way coupling between mesh and NeRF during rendering and simulation. We first review the light transport equations for both mesh and NeRF, then distill them into an efficient algorithm for updating radiance and throughput along a cast ray with an arbitrary number of bounces. To resolve the discrepancy between the linear color space that the path tracer assumes and the sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range (HDR) images. We also present a strategy to estimate light sources and cast shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric formulation can be efficiently integrated with a high-performance physics simulator that supports cloth, rigid and soft bodies. The full rendering and simulation system can be run on a GPU at interactive rates. We show that a hybrid system approach outperforms alternatives in visual realism for mesh insertion, because it allows realistic light transport from volumetric NeRF media onto surfaces, which affects the appearance of reflective/refractive surfaces and illumination of diffuse surfaces informed by the dynamic scene.