TACO：工具增强的智能体工具使用信用优化

摘要

具身多模态模型通过代码对图像执行多样化操作，并基于返回的视图进行推理，这成为细粒度视觉问答的有效范式。然而，代码操作可能是有用的、冗余的或具有误导性的。仅依赖结果奖励无法精确区分这些情况，而现有的过程奖励要么无法将最终正确性归因于单个工具调用，要么需要外部评判模型。为解决这一问题，我们提出工具增强信用优化（TACO），这是一种基于两个耦合优势通道的GRPO变体，专为代码工具代理设计。第一个通道是差分答案探测奖励（DAPR），这是一种自监督、无需评判模型的工具贡献优势，通过工具自身对正确回答的影响来评估每次调用的信用。在模型推理中插入探测标记，可分别获取工具存在与缺失时的预测结果，并将结果奖励的差值作为该调用的价值：有用调用为正，误导性调用为负，无影响调用为零。该方法复用了现有答案验证器，无需辅助评判模型，且由于采用差值而非绝对探测分数，自然能抵御探测攻击。第二个通道是通过结果门控优势路由（OGAR）分配的最终答案结果优势：这是一种无参数规则，根据工具调用的结果状态，仅将信用分发给负责的片段，从而在不引入任何成本项的前提下抑制无效工具调用。我们通过两阶段SFT+RL流程训练TACO。在感知、推理及通用多模态基准上的大量实验表明，该方法能持续提升准确率，并学会仅在必要时调用工具。

English

Agentic multimodal models perform diverse operations on an image via code and reason over the returned view, an effective paradigm for fine-grained visual question answering. However, code operations can be useful, redundant, or misleading. Outcome-only rewards cannot precisely distinguish these cases, and existing process rewards either fail to attribute final correctness to individual tool calls, or require an external judge model. To address this, we introduce Tool-Augmented Credit Optimization (TACO), a GRPO variant for code-tool agents built on two coupled advantage channels. The first, Differential Answer-Probe Reward (DAPR), is a self-supervised, judge-free tool-contribution advantage that credits each tool call by its own effect on answering correctly. Probe tokens inserted into the model's reasoning elicit its predictions with and without the tool, and the difference in outcome reward is taken as the call's value: positive for a useful call, negative for a misleading one, and zero for one that changes nothing. This reuses the existing answer checker with no auxiliary judge, and, being a difference rather than an absolute probe score, is naturally robust to probe-hacking. The second is the outcome advantage from the final answer, distributed by Outcome-Gated Advantage Routing (OGAR): a parameter-free rule that, conditioned on the call's outcome, delivers this credit only to the responsible segments, suppressing wasted tool calls without any cost term. We train TACO through a two-stage SFT+RL pipeline. Extensive experiments across perception, reasoning, and general multimodal benchmarks show that it yields consistent accuracy gains and learns to invoke its tools only when they help.