几何至关重要：学习语义对应的三维基础先验

摘要

来自自监督视觉模型和文本到图像扩散模型的基础特征已被证明对语义对应估计有效。然而，由于这些特征主要从二维图像目标中学习，它们缺乏明确的3D感知能力，常常混淆对称物体侧面、重复部件以及在3D中截然不同的视觉相似结构。我们提出了一种3D感知后训练框架，该框架通过融入3D基础模型的先验知识，超越了现有的二维基础特征。对于给定的图像，我们的方法利用SAM3D估计物体几何与姿态，并通过渲染-比较优化来细化姿态。随后，我们根据估计的物体姿态，将重建几何中的PartField描述子渲染到图像平面。由此产生的几何感知特征图补充了DINO和Stable Diffusion的特征，而重建形状上的测地距离则能可靠地筛选候选对应关系。我们使用筛选后的匹配作为监督信号，在DINO和Stable Diffusion之上训练一个轻量级适配器，用于语义对应。与以往需要姿态标注并依赖粗糙球面几何的后训练方法不同，我们的方法自动获取实例特定的3D结构，并利用其指导对应学习。实验表明，我们的方法在提升语义对应性能的同时，减少了人工几何监督。代码和模型可在 https://github.com/GenIntel/3D-SC 获取。

English

Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.