具有三維感知二維表示的潛在輻射場

Latent Radiance Fields with 3D-aware 2D Representations

February 13, 2025

作者: Chaoyi Zhou, Xi Liu, Feng Luo, Siyu Huang

cs.AI

摘要

潛在的3D重建已顯示出在賦予3D語義理解和3D生成方面的巨大潛力，通過將2D特徵提煉到3D空間中。然而，現有方法在2D特徵空間和3D表示之間存在領域差距，導致渲染性能下降。為應對這一挑戰，我們提出了一種新穎的框架，將3D意識融入2D潛在空間中。該框架包括三個階段：(1)一種考慮對應的自編碼方法，增強2D潛在表示的3D一致性，(2)一個潛在輻射場(LRF)，將這些具有3D意識的2D表示提升到3D空間，以及(3)一種VAE-輻射場(VAE-RF)對齊策略，改善從渲染的2D表示解碼的圖像。大量實驗表明，我們的方法在合成性能和跨不同室內外場景數據集的泛化能力方面優於最先進的潛在3D重建方法。據我們所知，這是首個展示從2D潛在表示構建的輻射場表示能夠產生逼真的3D重建性能的工作。

English

Latent 3D reconstruction has shown great promise in empowering 3D semantic understanding and 3D generation by distilling 2D features into the 3D space. However, existing approaches struggle with the domain gap between 2D feature space and 3D representations, resulting in degraded rendering performance. To address this challenge, we propose a novel framework that integrates 3D awareness into the 2D latent space. The framework consists of three stages: (1) a correspondence-aware autoencoding method that enhances the 3D consistency of 2D latent representations, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field (VAE-RF) alignment strategy that improves image decoding from the rendered 2D representations. Extensive experiments demonstrate that our method outperforms the state-of-the-art latent 3D reconstruction approaches in terms of synthesis performance and cross-dataset generalizability across diverse indoor and outdoor scenes. To our knowledge, this is the first work showing the radiance field representations constructed from 2D latent representations can yield photorealistic 3D reconstruction performance.