RenderFormer-V2：基於異質場景基元的神經渲染

摘要

我們提出「RenderFormer-V2」，一個統一的、基於學習式 Transformer 的神經渲染模型，可與現代基於物理的渲染系統互補；無需逐場景訓練或專用程式碼，即可處理多樣的光傳輸效應，例如焦散、體積散射、環境光照、具紋理與置換的表面，以及分佈外材質。RenderFormer-V2 將全域光傳輸建模為序列到序列轉換。承襲其前代模型，RenderFormer-V2 亦採用兩階段流程：一個視點無關階段，用於解算場景內基元對基元傳輸；以及一個視點相關階段，將內部神經場景表徵轉換為影像像素。與 RenderFormer 不同的是，我們的模型在視點無關階段採用一種新穎的機制，結合視窗化注意力與受渲染啟發的注意力匯點，以提升可擴展性，同時維持渲染準確度。為了進一步提升多功能性，RenderFormer-V2 支援異質場景基元，包括環境貼圖與參與介質，並採用一種獨立於底層表面反射模型的材質編碼，該編碼透過新穎的神經嵌入來編碼材質外觀。我們在各種場景上展示 RenderFormer-V2 的多功能性，並對改良後的注意力機制進行廣泛的消融實驗。

English

We present 'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism.