Agent-G^2：智能體強化學習的高斯引導

摘要

基於提示的強化學習透過在每次回合前保留專家軌跡的前綴，讓策略從更接近成功的狀態開始探索，藉此解決長程智慧體任務中的獎勵稀疏問題。其效果取決於引導深度：也就是要保留多少軌跡。現有方法將此深度視為確定性純量。排程式方法對所有樣本共用單一數值，忽略每個任務的異質性；逐樣本探測則以額外回合為代價，分別估計每個樣本的深度。我們發現，有用的引導落在一個深度區帶內，其資訊量輪廓在帶中心附近近似高斯分布，而非集中在單一最適點。我們提出 Agent-G^2，一個高斯引導框架，針對每個任務從高斯分布中抽取深度，該分布的中央與離散度是根據已收集用於策略最佳化的回合線上估計，不需額外探測回合或學習深度預測器。中心結合全域基線與每群集難度，離散度追蹤群集內變異數。我們在 ALFWorld 與 WebShop 上使用 Qwen2.5-1.5B / 7B-Instruct 評估 Agent-G^2。在不到逐樣本探測三分之一回合成本的情況下，Agent-G^2 在 ALFWorld 上比最強的提示式、無提示式與 Aux-RL 基線分別高出 2.3 / 3.9 / 7.4 分。

English

Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G^2, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G^2 on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G^2 outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.