視頻擴散模型用於手部動作重建的驚人有效性

摘要

從自我中心影片中進行四維手部動作重構，目前因現有方法的明顯限制而遭遇瓶頸：基於影像的流程依賴於在嚴重遮擋下會失效的檢測器，而基於影片的方法則依賴僅從稀少手部姿勢標註中學習到的時序模組，這是一種不足以建模動作動態、遮擋推理及手物互動的狹窄訊號。然而，這些能力正是影片生成模型在經由網路規模訓練以合成連貫影片時，必須隱含習得的。基於此，我們提出 ViDiHand，利用預訓練影片擴散模型的表徵來重構四維雙手姿勢。我們透過手部疊加渲染目標來調整它，使其特徵專注於手部，同時保留其世界先驗。接著，一個解碼器從調整後的特徵中恢復公制尺度的姿勢。整個流程直接作用於完整幀——無需檢測器、無需填充器、無需測試時最佳化。在 ARCTIC、HOT3D 和 HOI4D 資料集上，ViDiHand 大幅超越先前方法，將影片擴散模型確立為手部動作重構的強大新基礎，並為具身智慧提供一條邁向可擴展野外資料收集的可行途徑。專案頁面：https://vidihand.github.io。

English

4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.