关系视觉相似性

Relational Visual Similarity

December 8, 2025

作者: Thao Nguyen, Sicheng Mo, Krishna Kumar Singh, Yilin Wang, Jing Shi, Nicholas Kolkin, Eli Shechtman, Yong Jae Lee, Yuheng Li

cs.AI

摘要

人类不仅能看到属性相似性——还能识别关系相似性。苹果与桃子相似是因为二者都是红色水果，但地球也与桃子相似：地壳、地幔和地核分别对应桃子的表皮、果肉和果核。认知科学家认为，这种感知和识别关系相似性的能力正是人类区别于其他物种的关键特征。然而，当前广泛使用的视觉相似性度量方法（如LPIPS、CLIP、DINO）仅关注感知属性相似性，未能捕捉人类所感知的丰富且常出人意料的关系相似性。我们该如何超越图像的可见内容来捕捉其关系属性？如何让具有相同关系逻辑的图像在表征空间中更加接近？为解答这些问题，我们首先将关系图像相似性形式化为可量化问题：当两幅图像内部视觉元素之间的关系或功能相互对应时，即使其视觉属性不同，它们也具有关系相似性。随后我们构建了包含11.4万条图像-文本对的数据集，其中文本经过匿名化处理——描述场景底层的关系逻辑而非表面内容。利用该数据集，我们对视觉-语言模型进行微调以衡量图像间的关系相似性。该模型成为通过底层关系结构（而非表面视觉外观）连接图像的首步尝试。我们的研究表明，虽然关系相似性具有大量实际应用场景，但现有图像相似性模型均未能有效捕捉这一特性——这揭示了视觉计算领域的关键空白。

English

Humans do not just see attribute similarity -- we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its crust, mantle, and core correspond to the peach's skin, flesh, and pit. This ability to perceive and recognize relational similarity, is arguable by cognitive scientist to be what distinguishes humans from other species. Yet, all widely used visual similarity metrics today (e.g., LPIPS, CLIP, DINO) focus solely on perceptual attribute similarity and fail to capture the rich, often surprising relational similarities that humans perceive. How can we go beyond the visible content of an image to capture its relational properties? How can we bring images with the same relational logic closer together in representation space? To answer these questions, we first formulate relational image similarity as a measurable problem: two images are relationally similar when their internal relations or functions among visual elements correspond, even if their visual attributes differ. We then curate 114k image-caption dataset in which the captions are anonymized -- describing the underlying relational logic of the scene rather than its surface content. Using this dataset, we finetune a Vision-Language model to measure the relational similarity between images. This model serves as the first step toward connecting images by their underlying relational structure rather than their visible appearance. Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it -- revealing a critical gap in visual computing.