Harness-G：面向搜索代理的圖結構化控制框架

摘要

強化學習（RL）搜尋代理通常將檢索建模為自由形式的自然語言查詢生成，並使用最終答案獎勵來最佳化多輪互動。當前研究主要透過更密集或更有結構的信用訊號來改善訓練，但很少檢視檢索在策略-環境介面上是否被適當建構。我們觀察到 Search-R1 訓練期間存在顯著的檢索別名化：相同問題的展開軌跡持續生成不同的查詢字串，但它們累積的證據集卻日益重疊。我們將此現象稱為檢索等價崩潰；在此狀態下，軌跡在檢索決策方面趨近效用等價，導致組內回報缺乏有效的檢索對比。為了解決此問題，我們提出 Harness-G，一個重新設計此介面的圖結構檢索框架。它將自由形式的查詢生成重新建構為有限動作選擇：策略選擇一個證據句子或實體，或選擇回答，而環境負責建構選單、追蹤檢索狀態，並驗證與執行每個選擇。此介面減少了語言別名化，使同狀態下的替代方案可直接比較。在此介面基礎上，我們引入結構化非短視信用（SNC），它使用凍結的答案評分器將選定動作與其替代方案進行比較，並將下游收益分配給促成這些收益的較早動作。在六個問答基準上，Harness-G 在兩個評估的模型規模下均達成最高的平均 F1，在 1.5B 和 3B 規模下分別超越最強基線 Graph-R1 10.74 分和 3.98 分。

English

Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.