ObjectFolder基准测试：使用神经和真实对象进行多感官学习

The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects

June 1, 2023

作者: Ruohan Gao, Yiming Dou, Hao Li, Tanmay Agarwal, Jeannette Bohg, Yunzhu Li, Li Fei-Fei, Jiajun Wu

cs.AI

摘要

我们介绍了ObjectFolder Benchmark，这是一个包含10个任务的基准套件，用于多感官以物体为中心的学习，围绕着视觉、听觉和触觉的物体识别、重建和操作。我们还推出了ObjectFolder Real数据集，其中包括100个真实世界家用物品的多感官测量数据，构建在一个新设计的流程之上，用于收集真实物体的3D网格、视频、冲击声音和触觉读数。我们对来自ObjectFolder的1,000个多感官神经物体以及来自ObjectFolder Real的真实多感官数据进行了系统化基准测试。我们的结果表明多感官感知的重要性，并揭示了视觉、音频和触觉在不同以物体为中心的学习任务中的各自作用。通过公开发布我们的数据集和基准套件，我们希望在计算机视觉、机器人技术等领域推动并促进多感官以物体为中心的学习的新研究。项目页面：https://objectfolder.stanford.edu

English

We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulation with sight, sound, and touch. We also introduce the ObjectFolder Real dataset, including the multisensory measurements for 100 real-world household objects, building upon a newly designed pipeline for collecting the 3D meshes, videos, impact sounds, and tactile readings of real-world objects. We conduct systematic benchmarking on both the 1,000 multisensory neural objects from ObjectFolder, and the real multisensory data from ObjectFolder Real. Our results demonstrate the importance of multisensory perception and reveal the respective roles of vision, audio, and touch for different object-centric learning tasks. By publicly releasing our dataset and benchmark suite, we hope to catalyze and enable new research in multisensory object-centric learning in computer vision, robotics, and beyond. Project page: https://objectfolder.stanford.edu