PixelEyes：解耦感知与推理以实现精准视觉证据搜寻

摘要

本文探讨了多轮视觉推理问题，并观察到多模态大语言模型（MLLMs）在定位目标时反复失败，导致生成冗长且冗余的推理轨迹。我们将这种失败归因于推理与感知在同一模型中的耦合——MLLM同时进行推理和定位，而不准确的定位会触发额外的推理轮次，使得路径变得臃肿。针对这一问题，我们提出了PixelEyes，一种显式解耦推理与感知的多轮视觉推理智能体：推理器决定“寻找什么”，而专门的感知工具回答“在哪里”。具体来说，PixelEyes引入了以下设计：1）掩码引导的视觉搜索。调用指代分割模型提供精确到掩码的定位，使推理器无需补偿定位误差；2）语义区域广度优先搜索（BFS）。为避免反复裁剪错误子区域导致的冗余循环，我们将探索过程组织为对语义区域的广度优先搜索。为内化这些能力，我们基于现有数据重新合成专家轨迹，构建了PixelEyes-6K数据集，将掩码引导搜索和BFS逻辑显式嵌入模型中。此外，我们提出了Pinpoint-Bench——一个零提示视觉搜索基准（即问题中不提供任何位置线索），并提供实例级掩码和边界框，将定位失败与推理失败分离，从而实现对“无意盲视”等失败模式的细粒度分析。当前最先进的MLLMs和视觉推理智能体在Pinpoint-Bench上仍有较大提升空间，这验证了该基准的质量与难度。代码和模型均已开源。

English

This paper explores multi-turn visual reasoning and observes that MLLMs repeatedly fail to localize the target, leading to long, redundant trajectories. We attribute this failure to the entanglement of reasoning and perception within a single model, the MLLM reasons and localizes simultaneously, and inaccurate localization triggers additional reasoning turns that bloat the trajectory. To solve this problem, we propose PixelEyes, a multi-turn visual reasoning agent that explicitly decouples reasoning from perception, i.e., the reasoner decides what to look for, while a specialized perception tool answers where it is. Specifically, PixelEyes introduces 1) Mask-guided Visual Search. A referring segmentation model is invoked to provide mask-precise localization, freeing the reasoner from the need to compensate for imprecise grounding. 2) Semantic-region Breadth-first Search (BFS). To eliminate redundant loops caused by repeatedly cropping incorrect sub-regions, we organize exploration as a breadth-first search over semantic regions. To internalize these capabilities, we construct the PixelEyes-6K dataset by resynthesizing expert trajectories from existing data. This explicitly embeds our mask-guided search and BFS logic into the model. We further introduce Pinpoint-Bench, a zero-hint visual search benchmark, i.e., no location cues are provided in the question, with instance-level masks and bounding boxes that separate localization failures from reasoning failures, enabling fine-grained analysis of failure modes such as inattentional blindness. Recent state-of-the-art MLLMs and visual reasoning agents leave large headroom on Pinpoint-Bench, demonstrating its quality and difficulty. Code and models are open-sourced.