視覺工具使用的幻象：對圖像思考的因果審查

摘要

「以圖思考」（thinking-with-images）典範為多模態大型語言模型配備主動的視覺操作，例如裁剪與縮放。然而，使用這些操作的模型相較於直接推論，往往僅能獲得邊際甚至負面的增益，卻伴隨顯著更高的 token 成本。它們也可能重複裁剪無關區域，並在直接推論能正確回答的問題上失敗。我們追問：返回的視覺證據是否對答案產生因果效應？為回答此問題，我們將視覺工具使用形式化為一個因果圖，將觀測中介的路徑與動作誘發的捷徑區分開來。接著我們透過三個層級的干預對其進行審計：策略層級（比較工具使用與直接推論）、軌跡層級（在推演過程中破壞所有觀測），以及步驟層級（在固定前綴下反事實地替換單一觀測）。我們的步驟層級估計量——視覺證據增益（Visual Evidence Gain）——隔離了每個返回觀測的貢獻。在六個代表性模型與五個細粒度感知基準上，我們發現策略校準不良，存在兩種失敗模式。在「不看就呼叫」（Calling Without Looking）中，返回的觀測對答案沒有因果效應。在「不規劃就看」（Looking Without Planning）中，觀測具有資訊量，但呼叫排程不一致。軌跡層級的診斷將策略層級的準確率增益分解，並顯示增益集中在「校準良好的少數」上。我們將此差異稱為視覺工具使用的幻覺：儘管整體準確率有所提升，視覺工具使用在廣泛的推演範圍中並非因果有效地發揮作用。程式碼位於 https://github.com/OpenCausaLab/CauAudit。

English

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.