H2R-Grounder：一种无需配对数据的范式——将人类交互视频转化为物理实体机器人视频

H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos

December 10, 2025

作者: Hai Ci, Xiaokang Liu, Pei Yang, Yiren Song, Mike Zheng Shou

cs.AI

摘要

能够从日常人类视频中学习操作技能的机器人，无需繁琐的机器人数据收集即可获得广泛能力。我们提出一种视频到视频的转换框架，可将普通人机交互视频转化为具有真实物理交互效果且运动一致的机器人操作视频。该方法无需任何配对的人机视频进行训练，仅需一组非配对的机器人视频即可，使得系统易于扩展。我们引入一种可迁移的表征方式来弥合实体差异：通过修复训练视频中的机械臂以获得干净背景，并叠加简单视觉提示（标记点和箭头指示夹爪位置与方向），可让生成模型条件化地将机械臂重新插入场景。测试时，我们对人类视频实施相同流程（修复人体并叠加人体姿态提示），生成能模仿人类动作的高质量机器人视频。我们采用上下文学习方式对SOTA视频扩散模型（Wan 2.2）进行微调，确保时间连贯性并充分利用其丰富的先验知识。实证结果表明，相较于基线方法，我们的方法能实现显著更真实且具物理依据的机器人运动，为通过无标注人类视频扩展机器人学习指明了前景广阔的方向。项目页面：https://showlab.github.io/H2R-Grounder/

English

Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/