SimFoundry: 面向策略学习与评估的模块化与自动化场景生成

摘要

在现实世界中训练和评估机器人策略成本高昂且难以规模化。我们提出SimFoundry，一个模块化自动化系统，能够从视频零样本构建真实到仿真场景。SimFoundry可生成可直接用于仿真的数字孪生，并支持物体、场景和任务编辑，从而自动生成多样化的数字变体：对重构真实场景进行保持功能可供性的变异。基于SimFoundry数据训练的策略可零样本迁移至涉及多步操作、铰接物体交互及双臂交互等具有挑战性的现实任务，其数字变体（原始场景、物体和任务的变体）有助于泛化至新的现实环境条件。在7个操作任务和5种策略架构上，SimFoundry仿真评估对现实表现具有强预测能力，平均皮尔逊相关系数达0.911，最大排名违规均值仅0.018。当仿真训练策略在现实世界进行零样本评估时，使用物体、场景和任务变体进行训练的策略，其任务成功率平均分别提升17%、21%和40%。更多详情请见https://research.nvidia.com/labs/gear/simfoundry/。

English

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .