ReViV：從單眼第一人稱影片重建觀看者與視野的4D

摘要

自我中心设备（例如可穿戴式前置摄像头）提供了独特的视角，用于捕捉人类观察者与周围环境之间的连续交互。因此，一个能够重建这种4D表示的整体高效多模态模型极具吸引力。然而，现有方法通常依赖辅助输入（如预先计算的相机轨迹），将场景感知和人体自我运动建模视为独立问题（尽管它们之间存在强相互依赖关系），并且推理时间较慢。为了解决这些局限性，我们提出了ReViV，这是首个统一框架，用于从单目RGB视频中同时提取观察者与场景的动态，实现整体自我中心4D重建。我们将该任务定义为学习多模态信号的完整联合概率分布，包括RGB视频、相机轨迹、注视方向、全身运动、手部运动和深度。在掩码生成式自我中心Transformer的驱动下，ReViV采用单一前馈架构，以快速推理速度同时重建观察者和场景在时间上一致的4D表示。在多个基准数据集（包括HoloAssist、HOT3D、ARCTIC、Aria Digital Twin和TACO）上的大量实验表明，ReViV在整体自我身体、手部、注视重建以及相机跟踪方面实现了最先进的准确性和效率，同时在不依赖大量任务特定先验的情况下，保持了极具竞争力的自我中心深度估计性能。代码和模型已完全开源：https://reviv4d.github.io/。

English

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.