TACO：面向智能代理工具使用的工具增強型信用優化

摘要

代理型多模态模型能通过程式码对影像执行多种操作，并根据回传的视图进行推理，这是一种用于细粒度视觉问答的有效范式。然而，程式码操作可能是有益的、多余的，甚至具有误导性。仅基于结果的奖励无法精确区分这些情况，而现有的过程奖励要么无法将最终正确性归因于个别工具调用，要么需要外部评判模型。为解决此问题，我们提出了工具增强信用优化（TACO），这是一种针对程式码工具代理的GRPO变体，基于两条耦合的优势通道构建。第一个是差异式答案探针奖励（DAPR），这是一种自监督、无需评判模型的工具贡献优势，它根据每个工具调用自身对正确回答的影响来赋予其信用。在模型的推理中插入探针标记，可以分别在有工具和无工具的情况下引发出模型的预测，而结果奖励的差异则被视为该次调用的价值：对于有益调用为正、误导性调用为负、无变化调用为零。该方法重复使用了现有的答案检查器，无需辅助评判模型，而且由于是差异值而非绝对探针分数，因此本质上对探针欺骗具有鲁棒性。第二个优势来自最终答案的结果优势，通过结果门控优势路由（OGAR）进行分配：这是一种无参数的规则，根据调用的结果，仅将这一信用分配给负责的推理片段，从而在不引入任何成本项的情况下抑制无用的工具调用。我们通过两阶段的SFT+RL流水线训练TACO。在感知、推理和通用多模态基准上的大量实验表明，该方法能带来一致的准确率提升，并学会仅在工具有帮助时才进行调用。

English

Agentic multimodal models perform diverse operations on an image via code and reason over the returned view, an effective paradigm for fine-grained visual question answering. However, code operations can be useful, redundant, or misleading. Outcome-only rewards cannot precisely distinguish these cases, and existing process rewards either fail to attribute final correctness to individual tool calls, or require an external judge model. To address this, we introduce Tool-Augmented Credit Optimization (TACO), a GRPO variant for code-tool agents built on two coupled advantage channels. The first, Differential Answer-Probe Reward (DAPR), is a self-supervised, judge-free tool-contribution advantage that credits each tool call by its own effect on answering correctly. Probe tokens inserted into the model's reasoning elicit its predictions with and without the tool, and the difference in outcome reward is taken as the call's value: positive for a useful call, negative for a misleading one, and zero for one that changes nothing. This reuses the existing answer checker with no auxiliary judge, and, being a difference rather than an absolute probe score, is naturally robust to probe-hacking. The second is the outcome advantage from the final answer, distributed by Outcome-Gated Advantage Routing (OGAR): a parameter-free rule that, conditioned on the call's outcome, delivers this credit only to the responsible segments, suppressing wasted tool calls without any cost term. We train TACO through a two-stage SFT+RL pipeline. Extensive experiments across perception, reasoning, and general multimodal benchmarks show that it yields consistent accuracy gains and learns to invoke its tools only when they help.