SimFoundry：針對策略學習與評估的模組化自動場景生成

摘要

在现实世界中训练和评估机器人策略成本高昂且难以规模化。我们提出 SimFoundry——一个模块化自动化系统，能够从视频中实现零样本的实到仿场景构建。SimFoundry 可生成可直接用于仿真的数字孪生，并支持物体、场景及任务编辑，从而自动生成多样化的数字变体：即保留功能属性的重构真实世界场景变体。基于 SimFoundry 数据训练的策略可零样本迁移至涉及多步操作、铰接物体交互及双手协作等挑战性真实任务中；其数字变体（原始场景、物体及任务的变种）则有助于泛化至新的现实条件。在 7 项操作任务与 5 种策略架构的测试中，SimFoundry 的仿真评估能强有力地预测真实世界性能，平均皮尔逊相关系数为 0.911，最大排序违规均值仅 0.018。当在真实世界中零样本评估仿真训练的策略时，使用物体变体、场景变体及任务变体在仿真中训练的策略，其平均任务成功率分别提升 17%、21% 及 40%。更多详情请见 https://research.nvidia.com/labs/gear/simfoundry/ 。

English

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry generates sim-ready digital twins and supports object, scene, and task editing, enabling the automated generation of diverse digital cousins: affordance-preserving variations of reconstructed real-world scenes. Policies trained on SimFoundry data transfer zero-shot to challenging real tasks involving multi-step manipulation, articulated object interaction, and bimanual interaction, and its digital cousins (variations of the original scene, objects, and tasks) facilitate generalization to new real-world conditions. Across 7 manipulation tasks and 5 policy architectures, SimFoundry simulation evaluations strongly predict real-world performance, with mean Pearson correlation 0.911 and mean maximum ranking violation 0.018. When evaluating sim-trained policies zero-shot in the real world, policies trained with object, scene, and task cousins in simulation show average task success rate improvements of 17%, 21%, and 40%, respectively. Additional details at https://research.nvidia.com/labs/gear/simfoundry/ .