Harness-G：面向搜索智能体的图结构框架

摘要

强化学习（RL）搜索智能体通常将检索建模为自由形式的自然语言查询生成，并利用最终答案奖励来优化多轮交互。现有研究主要通过更密集或更结构化的信用信号来改进训练，但很少考察检索在策略-环境接口处是否得到了恰当的形式化表述。我们在Search-R1训练过程中观察到明显的检索混叠现象：同一问题的不同轨迹持续生成不同的查询字符串，但其累积的证据集却日益重叠。我们将这一现象称为检索等价坍缩；在此状态下，轨迹在检索决策上趋近于效用等价，使得组内回报缺乏有效的检索对比度。为解决这一问题，我们提出了Harness-G——一种重新设计该接口的图结构检索框架。它将自由形式查询生成重构为有限动作选择：策略选择一条证据句子或实体，或选择作答；环境则负责构建选项菜单、跟踪检索状态，并对每个选择进行验证和执行。该接口减少了语言混叠，并使同一状态下的备选动作之间可直接比较。在此接口基础上，我们引入了结构化非近视信用分配（Structured Non-myopic Credit, SNC），它使用冻结的答案评分器将所选动作与其备选动作进行比较，并将下游收益分配给促使这些收益实现的先前动作。在六个问答基准上，Harness-G在两种评估的模型规模下均取得了最高的平均F1值，分别比最强基线Graph-R1高出10.74个百分点（1.5B规模）和3.98个百分点（3B规模）。

English

Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.