CoVA-SFT：一個用於視覺抽象鏈的大規模資料集

摘要

思維鏈（Chain-of-thought, CoT）推理透過讓大型語言模型（LLMs）將問題分解為中間步驟，大幅提升了其能力。雖然 CoT 在語言任務上廣泛有效，但純文字的 CoT 迫使模型將視覺問題序列化為笨拙的文字表述。儘管已有架構性解決方案來處理視覺輸入，學術界仍缺乏一個大規模、多步驟、自我校正的資料集，用以教導模型在解決純文字推理問題時建立並維護內部視覺工作空間。為了解決此限制，我們提出了 CoVA-SFT，一個高度結構化的語料庫，包含 51.9K 個樣本，涵蓋 5 個不同的布局家族與 17 項複雜任務中的超過 222K 個多模態推理步驟；同時提出 CoVA-Bench，一個配套基準，包含 1,700 個涵蓋相同任務的保留測試樣本，以確保可重現的評估。藉由提供明確的推理依據表述、代理式渲染與驗證循環，CoVA-SFT 教導多模態語言模型將文字與視覺抽象交錯使用。我們驗證了該資料集，結果顯示在 CoVA-Bench 上，經 CoVA-SFT 微調的模型平均表現優於所有交錯式 CoT 基線 2 倍以上，但仍不及強大的純文字 CoT 基線，這突顯了未來工作的開放性挑戰。

English

Chain-of-thought (CoT) reasoning has dramatically improved large language models (LLMs) by allowing them to decompose problems into intermediate steps. While CoT is widely effective for linguistic tasks, text-only CoT forces models to serialize visual problems into awkward prose. Although architectural solutions exist to process visual inputs, the community lacks a massive, multi-step, self-corrected dataset to teach models how to build and maintain internal visual workspaces when solving purely textual reasoning problems. To address this limitation, we introduce CoVA-SFT, a highly structured corpus of 51.9K samples containing over 222K multimodal reasoning steps across 5 distinct layout families and 17 complex tasks, and CoVA-Bench, a companion benchmark of 1,700 held-out test samples spanning the same tasks for reproducible evaluation. By providing explicit rationale formulations, agentic renderings, and verification loops, CoVA-SFT teaches multimodal language models to interleave text and visual abstractions. We validate the dataset by demonstrating that models fine-tuned on CoVA-SFT outperform all interleaved CoT baselines by more than 2x on average on CoVA-Bench, though they still fall short of strong text-only CoT baselines, highlighting open challenges for future work.