RenderFormer：基于Transformer的三角形网格神经渲染与全局光照

RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination

May 28, 2025

作者: Chong Zeng, Yue Dong, Pieter Peers, Hongzhi Wu, Xin Tong

cs.AI

摘要

我们提出了RenderFormer，一种神经渲染管线，它能够直接从基于三角形的场景表示中渲染出包含完整全局光照效果的图像，且无需针对每个场景进行训练或微调。不同于传统的以物理为中心的渲染方法，我们将渲染过程建模为一个序列到序列的转换任务，即将代表带有反射属性的三角形序列转换为代表像素小块的输出序列。RenderFormer采用两阶段管线：第一阶段为视角无关阶段，负责建模三角形间的光传输；第二阶段为视角相关阶段，在视角无关阶段生成的三角形序列指导下，将代表光线束的标记转换为相应的像素值。这两个阶段均基于Transformer架构构建，并在最小先验约束下进行学习。我们在形状和光传输复杂度各异的场景上对RenderFormer进行了展示与评估。

English

We present RenderFormer, a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects and that does not require per-scene training or fine-tuning. Instead of taking a physics-centric approach to rendering, we formulate rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted to a sequence of output tokens representing small patches of pixels. RenderFormer follows a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the view-independent stage. Both stages are based on the transformer architecture and are learned with minimal prior constraints. We demonstrate and evaluate RenderFormer on scenes with varying complexity in shape and light transport.