CLBench-V：從基礎化到知識獲取的多模態上下文學習評估

摘要

真實世界的任務常要求模型從任務特定情境中學習，而非僅依賴預先訓練的知識。儘管近期研究已將此能力稱為情境學習，但現有評測主要聚焦於文字情境。然而，在許多實際應用中，需學習的情境其實是多模態的：科學發現透過圖表呈現、財務指標散佈於轉換報告中、空間決策則依賴地圖、場景或網頁。我們提出CLBench-V，一個專為多模態情境學習設計的基準，藉由圍繞三個維度組織任務來解決定位情境使用失效處的困難：情境基礎、新資訊應用與新知識學習。CLBench-V結合了轉換後的公開基準與新構建的資料集，涵蓋科學、金融、長文件理解、空間推理及網頁視覺問答等領域。為降低建構特定領域情境學習任務的成本，我們進一步對新資料集採用自動化建構與篩選流程。在3,443個實例與六個近期多模態模型上，最佳整體分數僅為0.2847，顯示多模態情境學習仍遠未飽和。此外，InternVL3.5-30B-A3B在情境基礎與新知識學習上表現最佳，而Qwen3.5-Plus在新資訊應用上則名列前茅。我們進一步分析了評判者可靠性、情境長度、圖片數量及代表性失敗案例。程式碼已於 https://github.com/IamLihua/CLBench-V 公開。

English

Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge. While recent work has highlighted this capability as context learning, existing evaluations mainly focus on textual contexts. In many practical settings, however, the context to be learned from is multimodal: scientific findings are conveyed through figures and tables, financial indicators are scattered across converted reports, and spatial decisions depend on maps, scenes, or web pages. We introduce CLBench-V, a benchmark for multimodal context learning that addresses the difficulty of localizing where context use breaks down by organizing tasks around three dimensions: context grounding, new information application, and new knowledge learning. CLBench-V combines converted public benchmarks with newly constructed datasets spanning domains such as science, finance, long-document understanding, spatial reasoning, and web-based visual question answering. To reduce the cost of constructing domain-specific context-learning tasks, we further use automated construction and filtering procedures for our newly built datasets. Across 3,443 instances and six recent multimodal models, the best overall score is only 0.2847, indicating that multimodal context learning remains far from saturated. Moreover, InternVL3.5-30B-A3B performs best on context grounding and new knowledge learning, while Qwen3.5-Plus performs best on new information application. We further analyze judge reliability, context length, image count, and representative failure cases. Code is available at https://github.com/IamLihua/CLBench-V.