Agent-G^2: 面向智能体强化学习的高斯引导

摘要

基于提示的强化学习通过在每个轨迹采样前保留专家轨迹的前缀，使策略能够从更接近成功的状态开始探索，从而解决长时程智能体任务中的奖励稀疏问题。其有效性取决于引导深度，即需要保留多少轨迹片段。现有方法将这一深度视为确定性标量。调度方法在所有样本间共享单一值，忽略了任务间的异质性；逐样本探测方法则单独估计每个样本的深度，但代价是额外的轨迹采样。我们发现，有效的引导信息存在于一个深度区间内，其信息量分布围绕区间中心呈近似高斯形状，而非集中于单一最优点。我们提出Agent-G²，一种高斯引导框架，对每个任务从高斯分布中采样深度，该分布的中心与宽度由策略优化过程中已收集的轨迹在线估计，无需探测轨迹或学习得到的深度预测器。分布中心结合了全局基线与逐簇难度，分布宽度则追踪簇内方差。我们在ALFWorld和WebShop上基于Qwen2.5-1.5B/7B-Instruct对Agent-G²进行了评估。Agent-G²在ALFWorld上以不到逐样本探测三分之一的轨迹采样成本，分别超出最强的基于提示、无提示和Aux-RL基线2.3/3.9/7.4个百分点。

English

Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G^2, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G^2 on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G^2 outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.