PixWorld: 在像素空间中统一三维场景生成与重建

摘要

3D重建与生成通常通过不同的范式来解决：基于像素的回归用于重建，潜在扩散用于生成。近期工作试图在潜在空间中统一两者，但存在显著缺陷：扩散目标定义在潜在特征上而非底层3D表示，且两个分支均受潜在编码引入的信息损失影响，同时需要预训练变分自编码器（VAE）或表示自编码器（RAE）。本文提出在统一的像素空间扩散范式下重新定义这两项任务，并引入PixWorld——一个同时处理3D重建与生成的单一模型。通过直接对渲染图像进行扩散监督，PixWorld消除了上述限制，使优化与3D场景保真度对齐。除了在2D图像层面运作且缺乏3D几何意识的光度与感知监督之外，我们进一步引入几何感知损失，将渲染视图与预训练3D基础模型的几何感知特征空间中的真实视图对齐，从而提供3D结构监督。PixWorld在性能上持续超越先前的潜在空间生成方法，并与最先进的重建方法相当，证明了统一像素空间方法的优越性。

English

3D reconstruction and generation are commonly tackled by separate paradigms: pixel-based regression for reconstruction, and latent diffusion for generation. Recent works attempt to unify them in latent space, but with notable drawbacks: the diffusion objective is defined on latent features rather than the underlying 3D representation, and both branches suffer from information loss introduced by latent encoding, while requiring a pretrained Variational Autoencoder (VAE) or Representation Autoencoder (RAE). In this paper, we reformulate these two tasks under a unified pixel-space diffusion paradigm and introduce PixWorld, a single model that jointly addresses 3D reconstruction and generation. By supervising diffusion directly on rendered images, PixWorld removes the above limitations and aligns optimization with 3D scene fidelity. Beyond photometric and perceptual supervision that operates at the 2D image level and lacks 3D geometric awareness, we further introduce a geometry perception loss that aligns rendered views with their ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision. PixWorld consistently outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods, demonstrating the superiority of a unified pixel-space approach.