ObjectFolder基準測試：利用神經網絡和真實物體進行多感官學習

The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects

June 1, 2023

作者: Ruohan Gao, Yiming Dou, Hao Li, Tanmay Agarwal, Jeannette Bohg, Yunzhu Li, Li Fei-Fei, Jiajun Wu

cs.AI

摘要

我們介紹了ObjectFolder Benchmark，這是一個包含10個任務的基準套件，用於多感官以物為中心的學習，圍繞著物體的識別、重建和視覺、聲音、觸覺的操作。我們還介紹了ObjectFolder Real數據集，其中包括100個現實世界家庭物品的多感官測量，基於一個新設計的流程，用於收集現實世界物體的3D網格、視頻、碰撞聲音和觸覺讀數。我們對來自ObjectFolder的1,000個多感官神經物體以及來自ObjectFolder Real的真實多感官數據進行系統化基準測試。我們的結果顯示了多感官知覺的重要性，並揭示了視覺、音頻和觸覺在不同以物為中心學習任務中的各自作用。通過公開發布我們的數據集和基準套件，我們希望在計算機視覺、機器人學等領域推動並促進多感官以物為中心學習的新研究。項目頁面：https://objectfolder.stanford.edu

English

We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulation with sight, sound, and touch. We also introduce the ObjectFolder Real dataset, including the multisensory measurements for 100 real-world household objects, building upon a newly designed pipeline for collecting the 3D meshes, videos, impact sounds, and tactile readings of real-world objects. We conduct systematic benchmarking on both the 1,000 multisensory neural objects from ObjectFolder, and the real multisensory data from ObjectFolder Real. Our results demonstrate the importance of multisensory perception and reveal the respective roles of vision, audio, and touch for different object-centric learning tasks. By publicly releasing our dataset and benchmark suite, we hope to catalyze and enable new research in multisensory object-centric learning in computer vision, robotics, and beyond. Project page: https://objectfolder.stanford.edu