具有三维感知的二维表示的潜在辐射场

Latent Radiance Fields with 3D-aware 2D Representations

February 13, 2025

作者: Chaoyi Zhou, Xi Liu, Feng Luo, Siyu Huang

cs.AI

摘要

潜在的3D重建展示了在将2D特征提炼到3D空间中，赋予3D语义理解和3D生成方面巨大潜力。然而，现有方法在2D特征空间和3D表示之间的领域差距方面存在困难，导致渲染性能下降。为了解决这一挑战，我们提出了一个将3D意识融入2D潜在空间的新颖框架。该框架包括三个阶段：（1）一种考虑对应关系的自动编码方法，增强了2D潜在表示的3D一致性，（2）一种潜在辐射场（LRF），将这些具有3D意识的2D表示提升到3D空间，以及（3）一种VAE-辐射场（VAE-RF）对齐策略，改善从渲染的2D表示解码图像。大量实验表明，我们的方法在合成性能和跨多样室内外场景数据集的泛化能力方面优于最先进的潜在3D重建方法。据我们所知，这是首个展示从2D潜在表示构建的辐射场表示能够产生逼真3D重建性能的工作。

English

Latent 3D reconstruction has shown great promise in empowering 3D semantic understanding and 3D generation by distilling 2D features into the 3D space. However, existing approaches struggle with the domain gap between 2D feature space and 3D representations, resulting in degraded rendering performance. To address this challenge, we propose a novel framework that integrates 3D awareness into the 2D latent space. The framework consists of three stages: (1) a correspondence-aware autoencoding method that enhances the 3D consistency of 2D latent representations, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field (VAE-RF) alignment strategy that improves image decoding from the rendered 2D representations. Extensive experiments demonstrate that our method outperforms the state-of-the-art latent 3D reconstruction approaches in terms of synthesis performance and cross-dataset generalizability across diverse indoor and outdoor scenes. To our knowledge, this is the first work showing the radiance field representations constructed from 2D latent representations can yield photorealistic 3D reconstruction performance.