點擊、輕拍、濺射：用於重建具挑戰性表面的觸覺資訊3D高斯濺射

Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces

March 29, 2024

作者: Mauro Comi, Alessio Tonioni, Max Yang, Jonathan Tremblay, Valts Blukis, Yijiong Lin, Nathan F. Lepora, Laurence Aitchison

cs.AI

摘要

觸覺和視覺緊密相關，相互增強我們理解世界的能力。從研究角度來看，混合觸覺和視覺的問題尚未得到充分探討，並提出了有趣的挑戰。為此，我們提出了Tactile-Informed 3DGS，這是一種新穎的方法，將觸覺數據（局部深度圖）與多視角視覺數據相結合，以實現表面重建和新視角合成。我們的方法通過優化3D高斯基元來準確地模擬物體在接觸點的幾何形狀。通過創建一個在接觸位置降低透射率的框架，我們實現了精緻的表面重建，確保深度圖均勻平滑。在考慮非蘭伯特物體（例如光滑或反射表面）時，觸覺特別有用，因為當代方法往往無法忠實重建反射高光。通過結合視覺和觸覺感知，我們比以往方法使用更少的圖像實現了更準確的幾何重建。我們對具有光澤和反射表面的物體進行評估，展示了我們方法的有效性，提供了重建質量的顯著改進。

English

Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.