LagerNVS：基于潜在几何的完全神经网络实时新视角合成

摘要

近期研究表明，神经网络无需显式三维重建即可完成新视角合成等三维任务。尽管如此，我们仍认为强三维归纳偏置对此类网络设计具有重要价值。为此我们提出LagerNVS——一种基于"三维感知"潜在特征的编解码器神经网络。该编码器由经过显式三维监督预训练的三维重建网络初始化，配合轻量级解码器，通过光度损失进行端到端训练。LagerNVS在确定性前馈新视角合成任务中（包含Re10k数据集上31.4 PSNR的表现）达到业界最优水平，无论相机参数是否已知均可实现实时渲染，对自然场景数据具有泛化能力，并能与扩散解码器结合实现生成式外推。

English

Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.

LagerNVS：基于潜在几何的完全神经网络实时新视角合成

LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis

摘要

Support