画你所见：多模态智能体中灵巧视觉工具使用的基准评测

摘要

评估正在从静态问答转向智能体环境，其中模型通过外部工具进行交互。我们识别出该领域中一个关键但尚未充分探索的能力——灵巧视觉工具使用：精细粒度、闭环的参数化视觉动作，模型从视觉证据中推断工具参数，而这些参数直接决定最终结果。现有基准涵盖网页导航、图形界面操作和软件工程，但很少针对视觉证据与执行精度之间的这种耦合关系。我们提出EASEL，一个评估灵巧视觉工具使用受控实例的基准，采用参考引导的视觉重建作为其主要代理任务：智能体逐步绘制画布以匹配参考图像。EASEL还包含涵盖区域标注、手写和路径规划的语义任务。我们进一步提供EASEL-Data，一个包含44万样本的两阶段课程式数据集用于轨迹监督，以及EASEL-9B以研究其对该能力的影响。对25个模型的评估显示，当前多模态智能体在EASEL上系统性表现不佳。重建相似度瓶颈处于低水平（0.40-0.54），而轨迹诊断揭示了严重的闭环不稳定性——模型通常在早期饱和或在峰值后退化。语义任务揭示了精确标注和路径规划方面的尖锐能力边界。EASEL-9B在EASEL-Data上训练后，超越基础模型相对6.3%，在所有被评估模型中排名第三。

English

Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.