ReViV：从单目第一人称视频中重建观察者与视图的四维空间

摘要

自我中心设备（如可穿戴前置摄像头）为捕捉人类观察者与周围环境之间的持续交互提供了独特视角。因此，一个能够重建这种四维表征的整体且高效的多模态模型极具需求。然而现有方法通常依赖辅助输入（如预计算的相机轨迹），将场景感知和人体自我运动建模视为独立问题（尽管二者高度依赖），且推理速度缓慢。为解决这些局限，我们提出ReViV——首个统一框架，用于从单目RGB视频中同时提取观察者与场景动态的整体自我中心四维重建。我们将该任务建模为学习多模态信号的完整联合概率分布，包括RGB视频、相机轨迹、注视方向、全身运动、手部运动及深度。借助掩码生成式自我中心变换器，ReViV采用单一前馈架构，以快速推理速度同步重建观察者与场景的时间一致性四维表征。在HoloAssist、HOT3D、ARCTIC、Aria Digital Twin和TACO等多个基准上的大量实验表明，ReViV在整体自我身体、手部、注视重建及相机追踪方面实现了最先进的精度与效率，同时在无需强任务特定先验的情况下保持了极具竞争力的自我中心深度估计性能。代码与模型已完全开源：https://reviv4d.github.io/。

English

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.