SoftVTBench：面向可變形物體操作的變形感知視觸覺數據集與基準

摘要

物理互動品質對於可變形物體操作至關重要，然而多數基準僅以任務成功與否作為評估標準。策略可能在容許滑移或造成過度擠壓的情況下完成任務。其主要瓶頸在於缺乏將策略可觀測的接觸觀測與完整任務過程中的獨立物理真實基準相互配對的視觸覺資料集。我們提出 SoftVTBench，一個針對物理互動感知之可變形物體操作所設計的視觸覺資料集。該資料集包含 4,000 筆專家示範及超過 50 個資產，涵蓋體積可變形物體與視覺匹配的剛體孿生。在 20 Hz 的頻率下，每個回合同步記錄多視角 RGB、雙指觸覺 RGB 與標記運動、本體感覺、語言描述、二元及連續夾爪動作，以及僅供評估者使用的有限元素狀態。在此資料集基礎上，我們建立了一個閉環基準，透過固定的物體特定校準來定義變形感知成功率（DSR），僅當 rollout 完成任務且其峰值正規化變形維持在容許範圍內時，才將其計為成功。在 Diffusion Policy、π₀.₅ 與 FastWAM 中，全部 12 個分佈內配置均包含違反變形容許範圍但仍成功的 rollout，佔各配置成功案例的 0.7% 至 24%。在分佈偏移下，視觸覺變體在六項策略—套件比較中皆達到較高的任務成功率，並在其中五項中達到較高的 DSR，而其分佈內的效益則好壞參半。這些結果顯示，提供觸覺資訊本身並不足以確保有效的多模態融合。因此，SoftVTBench 提供了一個共通的視觸覺資源，不僅可研究策略是否成功，更可探討策略如何與可變形物體進行物理互動，以及觸覺在何時能改善此類互動。

English

Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the task while allowing slip or causing excessive compression. A primary bottleneck is the absence of visuo-tactile datasets that pair policy-visible contact observations with independent physical ground truth over complete tasks. We introduce SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation. It contains 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions, alongside evaluator-only finite-element (FEM) states. Building upon this dataset, we establish a closed-loop benchmark that uses fixed object-specific calibration to define the Deformation-aware Success Rate (DSR), which counts a rollout as successful only when it completes the task and keeps peak normalized deformation within tolerance. Across Diffusion Policy, π_{0.5}, and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the deformation tolerance, accounting for 0.7--24% of each configuration's successes. Under distribution shift, visuo-tactile variants achieve higher task success in all six policy--suite comparisons and higher DSR in five, whereas their in-distribution benefits are mixed. These results show that making touch available does not by itself ensure effective multimodal fusion. SoftVTBench therefore provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.