手部运动重建中视频扩散模型的惊人有效性

摘要

从自我中心视频中进行4D手部运动重建受到现有方法的明显限制：基于图像的管线依赖在严重遮挡下失效的检测器，而基于视频的方法则依赖仅从稀缺的手部姿态标注中学习的时间模块，这些标注信号过于狭窄，不足以建模运动动态、遮挡推理和手物交互。然而，这些能力恰恰是视频生成模型在互联网规模上训练合成连贯视频时必须隐式获得的。受此启发，我们提出ViDiHand，利用预训练视频扩散模型的表征来重建4D双手姿态。我们通过手部叠加渲染目标来调整该模型，使其特征专门化于手部，同时保留其世界先验。随后，一个解码器从调整后的特征中恢复公制尺度的姿态。整个管线直接在完整帧上运行——无需检测器、无需填充器、无需测试时优化。在ARCTIC、HOT3D和HOI4D数据集上，ViDiHand显著优于先前方法，将视频扩散模型确立为手部运动重建的强大新基础，并为具身AI的可扩展野外数据收集提供了有前景的路径。项目页面：https://vidihand.github.io。

English

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.