後訓練中被忽視的免費午餐：LLM智能體的進步優勢

摘要

過程獎勵模型能對大型語言模型進行細粒度、逐步層級的評估，然而在智能體場景中建構此類模型仍極度困難：長程互動、不可逆動作以及隨機環境反饋，使得大規模人工標註與蒙地卡羅估計皆不可行。在本研究中，我們證明強化學習後訓練已提供進行有效逐步評分所需的要素，完全無需另外訓練專屬的獎勵模型。具體而言，我們在一般隨機馬可夫決策過程中推導出一種隱含優勢，稱之為「進展優勢」——經強化學習訓練的策略與其參考策略之間的對數機率比值恰好還原了最優優勢函數。此公式使產出的訊號無需標註、與領域無關，且可作為標準強化學習後訓練流程的副產品取得。我們在五個基準測試及四個模型家族上，針對三種不同應用（測試時擴展、不確定性量化與失敗歸因）驗證了進展優勢的有效性。在所有設定中，它一致優於基於信賴度的基準方法，並且雖然無需針對特定任務進行訓練，仍超越了專屬訓練的獎勵模型。我們透過對進展優勢特性的深入分析來補充這些結果，為其在真實世界智能體系統中的採用提供實務指引。

English

Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.