H2R-Grounder：一种无需配对数据的范式，实现人类交互视频向物理实体机器人视频的转化

H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos

December 10, 2025

作者: Hai Ci, Xiaokang Liu, Pei Yang, Yiren Song, Mike Zheng Shou

cs.AI

摘要

通过从日常人类视频中学习操作技能，机器人无需繁琐的数据采集即可获得广泛能力。我们提出一种视频到视频的转换框架，能将普通人机交互视频转化为具有真实物理交互效果且运动连贯的机器人操作视频。该方法仅需一组非配对的机器人视频进行训练，无需任何人机配对视频，使系统易于扩展。我们引入一种可迁移的表征方式来弥合本体差异：通过修复训练视频中的机械臂获得干净背景，并叠加简单视觉提示（指示夹爪位置与方向的标记和箭头），可引导生成模型将机械臂重新插入场景。测试时，我们对人类视频实施相同流程（修复人体并叠加姿态提示），生成能模仿人类动作的高质量机器人视频。我们采用上下文学习方式对SOTA视频扩散模型（Wan 2.2）进行微调，确保时间连贯性并利用其丰富的先验知识。实验结果表明，相较于基线方法，我们的方案能实现显著更真实且符合物理规律的机器人运动，为通过无标注人类视频扩展机器人学习指明了前景。项目页面：https://showlab.github.io/H2R-Grounder/

English

Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/