Deform360：用于可变形世界模型的大规模多视角视触觉数据集

摘要

预测物体动力学（即世界建模）是机器人操作领域的一项基本挑战，而可变形物体的建模因其高维状态空间和复杂的材料特性尤为困难。当前的世界模型主要通过两种不同范式来解决这一问题：在二维像素空间或更明确的三维几何空间上学习动力学。然而，由于缺乏多样化、大规模的真实世界数据，对这两种范式的相对优势与局限性的系统性认识仍不清晰。为解决这一问题，我们提出了 Deform360——一个大规模视觉触觉数据集，包含 198 个日常物品、1,980 个交互序列以及来自 41 台环视相机和双手触觉夹爪的超过 215 小时观测数据，用于捕捉整体运动与接触引发的局部形变。借助一种新颖的无标记视觉触觉 3D 追踪管线提取密集几何与运动信息，我们系统评估了当前最先进的世界模型，对二维视频模型与三维粒子模型进行了比较。最后，我们通过执行可变形物体上的机器人规划任务，初步展示了该数据集在真实世界中的适用性。我们的分析揭示了结构先验与可扩展性之间的关键权衡，为未来面向可变形物体的可泛化世界建模研究提供了坚实的基准。项目网站：https://deform360.lhy.xyz

English

Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz