CAFE：自我改进的搜索智能体需要协同进化的反馈

摘要

结果监督的搜索智能体能够学习何时以及如何检索证据，但最终奖励既无法定位中间错误，也无法在这些错误累积之前引导正在进行的轨迹。将纠正性反馈视为一种学习到的轨迹内干预，将两个角色耦合在一起：智能体必须决定何时请求并使用反馈，而评论者必须从结果混杂的轨迹展开中推断出有用的纠正——这些轨迹的失败模式会随着智能体的改进而变化。我们提出了CAFE（耦合智能体-反馈演化，Coupled Agent--Feedback Evolution）框架，其中共享参数模型在搜索智能体与评论者角色之间交替切换。CAFE从围绕基础智能体自身失败构建的轨迹中初始化反馈条件恢复，然后将在线与离线优化耦合起来。在线强化学习阶段，对比式反馈估计利用提示级别的调用-跳过成功差距来塑形请求回报，同时反馈感知优势塑形在反馈前后重新加权词元优势。离线阶段，轨迹衍生的偏好优化从配对的成功与不成功轨迹中学习反馈。在七个智能体搜索基准测试中，CAFE在平均表现上优于所评估的基于强化学习的搜索智能体，在全部六个领域外基准测试中保持其优势，并减少了答案级幻觉。单侧消融实验表明，仅改进智能体或仅改进评论者最终都会陷入平台期，而交替进行两种更新则持续提升性能。这些发现表明，自我改进的搜索智能体需要与其所引导的策略共同演化的反馈。

English

Outcome-supervised search agents learn when and how to retrieve evidence, but terminal rewards neither localize intermediate errors nor redirect an ongoing trajectory before those errors compound. Treating corrective feedback as a learned in-trajectory intervention couples the two roles: the agent must decide when to request and use feedback, while the critic must infer useful corrections from outcome-confounded rollouts whose failure patterns shift as the agent improves. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.