视觉工具使用的幻觉：图像思维的因果审计

摘要

"用图像思考"范式赋予多模态大语言模型主动视觉操作能力，如裁剪与缩放。然而，使用这些操作的模型相较于直接推理往往只获得边际增益甚至负增益，却付出高得多的词元成本。它们还可能反复裁剪无关区域，并在直接推理能正确回答的问题上失败。我们探究返回的视觉证据是否对答案产生因果影响。为回答这一问题，我们将视觉工具使用形式化为一个因果图，将观测中介路径与动作诱导捷径区分开来。随后，我们通过三个层面的干预对其进行审计：策略层面（比较工具使用与直接推理）、轨迹层面（在推演中破坏所有观测）以及步骤层面（在固定前缀下反事实地替换单个观测）。我们提出的步骤级估计目标——视觉证据增益——分离出每个返回观测的贡献。在六个代表性模型和五个细粒度感知基准上，我们揭示了策略失准的两种失败模式：在"不看就调用"中，返回的观测对答案没有因果影响；在"不规划就看"中，观测具有信息量，但调用调度不连贯。轨迹层面的诊断将策略层面的准确率增益分解，表明增益集中在少数校准良好的样本上。我们将这一差异称为视觉工具使用的幻觉：尽管总体准确率有所提升，视觉工具使用在广泛的推演范围中并非因果有效。代码见 https://github.com/OpenCausaLab/CauAudit。

English

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.