抓取、轻触、喷溅：基于触觉的3D高斯喷溅技术用于重建复杂表面

Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces

March 29, 2024

作者: Mauro Comi, Alessio Tonioni, Max Yang, Jonathan Tremblay, Valts Blukis, Yijiong Lin, Nathan F. Lepora, Laurence Aitchison

cs.AI

摘要

触觉与视觉相辅相成，共同提升我们对世界的理解能力。从研究角度来看，触觉与视觉的融合问题尚未得到充分探索，且充满有趣的挑战。为此，我们提出了Tactile-Informed 3DGS，这是一种创新方法，将触觉数据（局部深度图）与多视角视觉数据结合，以实现表面重建和新视角合成。我们的方法优化了三维高斯基元，以精确建模物体接触点的几何形状。通过构建一个在触点处降低透射率的框架，我们实现了精细的表面重建，确保深度图均匀平滑。在处理非朗伯体物体（如光亮或反射表面）时，触觉尤为有用，因为当代方法往往难以忠实重建镜面高光。通过结合视觉与触觉感知，我们以比以往方法更少的图像实现了更精确的几何重建。我们在具有光泽和反射表面的物体上进行了评估，并展示了我们方法的有效性，显著提升了重建质量。

English

Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.