PixelEyes：解耦感知與推理以精準搜尋視覺證據

摘要

本文探讨了多轮视觉推理问题，并观察到多模态大语言模型（MLLMs）会反复无法准确定位目标，从而导致冗长且重复的推理轨迹。我们将其归因于推理与感知在同一模型中的纠缠——MLLMs在推理的同时进行定位，而定位的不准确会触发额外的推理轮次，使得轨迹变得臃肿。为解决这一问题，我们提出了PixelEyes，一种将推理与感知明确解耦的多轮视觉推理智能体：即推理器决定“寻找什么”，而专门的感知工具回答“它在哪”。具体而言，PixelEyes引入了以下两个关键机制：1）掩码引导的视觉搜索：调用指代分割模型提供精确到掩码级别的定位，使推理器无需再为补偿不精确的指代而额外消耗能力；2）语义区域广度优先搜索（BFS）：为消除因反复裁剪错误子区域而导致的冗余循环，我们将探索过程组织为基于语义区域的广度优先搜索。为了将这些能力内化到模型中，我们通过从现有数据中重新合成专家轨迹，构建了PixelEyes-6K数据集，将掩码引导搜索和BFS逻辑显式嵌入其中。此外，我们还提出了Pinpoint基准（Pinpoint-Bench），这是一个零提示的视觉搜索基准——即问题中不提供任何位置线索，并配有实例级掩码和边界框，可将定位失败与推理失败区分开，从而实现对注意力盲视等失败模式的细粒度分析。当前最先进的多模态大语言模型和视觉推理智能体在Pinpoint基准上仍有大量提升空间，这充分证明了该基准的质量与难度。代码和模型均已开源。

English

This paper explores multi-turn visual reasoning and observes that MLLMs repeatedly fail to localize the target, leading to long, redundant trajectories. We attribute this failure to the entanglement of reasoning and perception within a single model, the MLLM reasons and localizes simultaneously, and inaccurate localization triggers additional reasoning turns that bloat the trajectory. To solve this problem, we propose PixelEyes, a multi-turn visual reasoning agent that explicitly decouples reasoning from perception, i.e., the reasoner decides what to look for, while a specialized perception tool answers where it is. Specifically, PixelEyes introduces 1) Mask-guided Visual Search. A referring segmentation model is invoked to provide mask-precise localization, freeing the reasoner from the need to compensate for imprecise grounding. 2) Semantic-region Breadth-first Search (BFS). To eliminate redundant loops caused by repeatedly cropping incorrect sub-regions, we organize exploration as a breadth-first search over semantic regions. To internalize these capabilities, we construct the PixelEyes-6K dataset by resynthesizing expert trajectories from existing data. This explicitly embeds our mask-guided search and BFS logic into the model. We further introduce Pinpoint-Bench, a zero-hint visual search benchmark, i.e., no location cues are provided in the question, with instance-level masks and bounding boxes that separate localization failures from reasoning failures, enabling fine-grained analysis of failure modes such as inattentional blindness. Recent state-of-the-art MLLMs and visual reasoning agents leave large headroom on Pinpoint-Bench, demonstrating its quality and difficulty. Code and models are open-sourced.