发现并利用斯佩尔克片段

Discovering and using Spelke segments

July 21, 2025

作者: Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen, Seungwoo Kim, Luca Thomas Wheeler, Jared Watrous, Ashley Xu, Gia Ancone, Wanhee Lee, Honglin Chen, Daniel Bear, Stefan Stojanov, Daniel Yamins

cs.AI

摘要

在计算机视觉领域，图像分割通常依据语义考量进行定义，并高度依赖于特定类别的惯例。相比之下，发展心理学研究表明，人类是以斯佩尔克物体（Spelke objects）的视角来感知世界的——这些物体是由物理实体组成的集合，在受到物理力作用时能够可靠地一起移动。因此，斯佩尔克物体基于与类别无关的因果运动关系运作，这可能更有利于支持诸如操作和规划等任务。本文首先对斯佩尔克物体概念进行了基准测试，引入了SpelkeBench数据集，该数据集包含了自然图像中多种定义明确的斯佩尔克分割。接着，为了从图像中算法化地提取斯佩尔克分割，我们构建了SpelkeNet，这是一类视觉世界模型，旨在预测未来运动的分布。SpelkeNet支持估计斯佩尔克物体发现的两个关键概念：(1) 运动可能性图，识别在“戳”动作下可能移动的区域；(2) 预期位移图，捕捉场景其余部分将如何移动。这些概念被用于“统计反事实探测”，即在具有高运动可能性的区域施加多样化的“虚拟戳”，并利用由此产生的预期位移图将斯佩尔克分割定义为相关运动统计的统计聚合。我们发现，在SpelkeBench上，SpelkeNet的表现优于如SegmentAnything（SAM）等有监督基线模型。最后，我们展示了斯佩尔克概念在实际应用中的实用性，当将其应用于多种现成的物体操作模型时，在3DEditBench物理物体操作基准测试中取得了更优的性能。

English

Segments in computer vision are often defined by semantic considerations and are highly dependent on category-specific conventions. In contrast, developmental psychology suggests that humans perceive the world in terms of Spelke objects--groupings of physical things that reliably move together when acted on by physical forces. Spelke objects thus operate on category-agnostic causal motion relationships which potentially better support tasks like manipulation and planning. In this paper, we first benchmark the Spelke object concept, introducing the SpelkeBench dataset that contains a wide variety of well-defined Spelke segments in natural images. Next, to extract Spelke segments from images algorithmically, we build SpelkeNet, a class of visual world models trained to predict distributions over future motions. SpelkeNet supports estimation of two key concepts for Spelke object discovery: (1) the motion affordance map, identifying regions likely to move under a poke, and (2) the expected-displacement map, capturing how the rest of the scene will move. These concepts are used for "statistical counterfactual probing", where diverse "virtual pokes" are applied on regions of high motion-affordance, and the resultant expected displacement maps are used define Spelke segments as statistical aggregates of correlated motion statistics. We find that SpelkeNet outperforms supervised baselines like SegmentAnything (SAM) on SpelkeBench. Finally, we show that the Spelke concept is practically useful for downstream applications, yielding superior performance on the 3DEditBench benchmark for physical object manipulation when used in a variety of off-the-shelf object manipulation models.