淺看深思：多模態思維鏈推理的能與不能

摘要

思維鏈（Chain-of-Thought, CoT）已成為透過引導逐步思考來提升大型語言模型（LLMs）推理能力的標準方法，但其在多模態任務中的有效性仍不明確。本文旨在系統性探討核心問題：多模態思維鏈推理能做什麼？它在哪些方面、為何存在不足？為此，我們在感知與推理兩大類別中評估了12項多模態任務，使用了14個非推理模型與8個推理模型。分析揭示了幾項重要發現：（1）CoT並非免費午餐，應根據各任務的具體需求選擇性使用。在感知任務中，CoT可能導致不良副作用，例如在視覺定位與物體計數上表現下降；反之，在涉及數學、科學及多圖像推理的推理任務中，CoT則證明有效；（2）與原始模型相比，現有開源多模態推理模型往往僅帶來微小的整體改進，這可能源於過度強調數學推理而犧牲了更廣泛的能力；（3）視覺推理仍是當前多模態CoT的主要瓶頸，模型呈現出「看輕、想重」的模式——在推理過程中，語言反思會起伏波動，而視覺反思則持續減弱。這些發現表明，儘管多模態CoT能相對妥善地處理語言反思，但它缺乏在整個推理過程中維持深度視覺內省的能力。

English

Chain-of-Thought (CoT) has become a standard method for improving reasoning capabilities in large language models (LLMs) by eliciting step-by-step thinking, but its effectiveness in multimodal tasks remains unclear. In this paper, we aim to systematically investigate the key question: What can multimodal Chain-of-Thought reasoning do, and where and why does it fall short? To this end, we evaluate 12 multimodal tasks across perception and reasoning categories using both 14 non-reasoning models and 8 reasoning models. Our analysis reveals several important findings: (1) CoT is not a free lunch and should be used selectively depending on the specific requirements of each task. For perception tasks, CoT can lead to undesirable side effects, such as reduced performance in visual grounding and object counting. In contrast, it proves effective for reasoning tasks involving mathematical, scientific, and multi-image reasoning; (2) Compared to original models, existing open-source multimodal reasoning models often yield only marginal overall improvements, possibly due to an overemphasis on mathematical reasoning at the expense of broader capabilities; (3) Visual reasoning remains a key bottleneck for current multimodal CoT, as models exhibit a Look Light, Think Heavy pattern where verbal reflection rises and falls during reasoning, whereas visual reflection consistently diminishes. These findings suggest that while multimodal CoT handles verbal reflection relatively well, it lacks the ability to maintain deep visual introspection throughout the reasoning process.