CAFE：自我改進的搜尋代理需要協同演化的回饋

摘要

以結果監督的搜尋代理學會何時及如何檢索證據，但最終獎勵既無法定位中間錯誤，也無法在這些錯誤累積之前引導進行中的軌跡。將矯正性回饋視為一種在軌跡內學習的介入，結合了兩個角色：代理必須決定何時請求並使用回饋，而評論者必須從受到結果混淆的展開中推斷有用的修正，這些展開的失敗模式會隨著代理的進步而改變。我們提出 CAFE（耦合代理－回饋演化），這是一個共享參數模型在搜尋代理與評論者角色之間交替的框架。CAFE 從圍繞基礎代理自身失敗所建立的軌跡中初始化以回饋為條件的恢復，然後耦合線上與離線最佳化。在線上強化學習期間，比較式回饋估計利用提示層級的呼叫－跳過成功差距來塑形請求回報，而回饋感知的優勢塑形則在回饋前後重新加權詞元優勢。在離線部分，從展開衍生的偏好最佳化從配對的成功與不成功軌跡中學習回饋。在七個代理式搜尋基準上，CAFE 平均表現優於所評估的基於強化學習的搜尋代理，在所有六個跨域基準上保持其優勢，並減少答案層級的幻覺。單側消融顯示，僅改善代理或僅改善評論者最終都會趨於平緩，而交替進行這兩種更新則持續提升效能。這些發現表明，自我改進的搜尋代理需要與其所引導的策略共同演化的回饋。

English

Outcome-supervised search agents learn when and how to retrieve evidence, but terminal rewards neither localize intermediate errors nor redirect an ongoing trajectory before those errors compound. Treating corrective feedback as a learned in-trajectory intervention couples the two roles: the agent must decide when to request and use feedback, while the critic must infer useful corrections from outcome-confounded rollouts whose failure patterns shift as the agent improves. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.