動態網格感知輻射場

摘要

將多邊形網格資產嵌入逼真的神經輻射場（NeRF）體積中，以便能夠以與 NeRF 物理一致的方式渲染它們並模擬其動態，這方面從將 NeRF 整合到傳統圖形管線的系統角度來看，尚未得到充分探討。本文設計了在渲染和模擬期間網格和 NeRF 之間的雙向耦合。我們首先回顧了網格和 NeRF 的光傳輸方程式，然後將它們提煉成一種有效的算法，用於沿著具有任意反射次數的投射射線更新輻射和通量。為了解決路徑追踪器假定的線性顏色空間與標準 NeRF 使用的 sRGB 顏色空間之間的差異，我們使用高動態範圍（HDR）圖像來訓練 NeRF。我們還提出了一種估計光源並在 NeRF 上投射陰影的策略。最後，我們考慮了如何將混合表面-體積形式與支持布料、剛體和軟體的高性能物理模擬器有效地整合。完整的渲染和模擬系統可以在 GPU 上以互動速率運行。我們展示了混合系統方法在網格插入的視覺逼真度方面優於其他方法，因為它允許體積 NeRF 媒體中的逼真光傳輸影響表面，這影響了反射/折射表面的外觀以及由動態場景資訊影響的漫反射表面的照明。

English

Embedding polygonal mesh assets within photorealistic Neural Radience Fields (NeRF) volumes, such that they can be rendered and their dynamics simulated in a physically consistent manner with the NeRF, is under-explored from the system perspective of integrating NeRF into the traditional graphics pipeline. This paper designs a two-way coupling between mesh and NeRF during rendering and simulation. We first review the light transport equations for both mesh and NeRF, then distill them into an efficient algorithm for updating radiance and throughput along a cast ray with an arbitrary number of bounces. To resolve the discrepancy between the linear color space that the path tracer assumes and the sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range (HDR) images. We also present a strategy to estimate light sources and cast shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric formulation can be efficiently integrated with a high-performance physics simulator that supports cloth, rigid and soft bodies. The full rendering and simulation system can be run on a GPU at interactive rates. We show that a hybrid system approach outperforms alternatives in visual realism for mesh insertion, because it allows realistic light transport from volumetric NeRF media onto surfaces, which affects the appearance of reflective/refractive surfaces and illumination of diffuse surfaces informed by the dynamic scene.

動態網格感知輻射場

Dynamic Mesh-Aware Radiance Fields

摘要

Support