QVal：低成本評估長時段LLM代理的密集監督訊號

摘要

大型語言模型代理日益在長期任務中運作，單一軌跡可能包含數百甚至數千個動作。在這些情境下，僅基於最終結果的獎勵所提供的指導過於稀疏，無法告知模型中間動作的優劣。密集監督方法旨在透過對中間步驟進行評分來解決此問題，這些評分方式涵蓋從內在信心、自蒸餾到嵌入相似度等範疇。然而，常見的評估方式是透過整合這些方法的訓練管線的下游表現來衡量其效能。這種做法成本高昂，會將監督品質與訓練工程的干擾變數混為一談，並使得需要不同訓練設定的不同方法論家族之間無法進行比較。結果，密集監督方法鮮少在共同基準上進行評測。我們提出 QVal，這是一個無需訓練的測試平台，用於直接評估密集監督信號。對於給定的狀態-動作對，QVal 衡量某方法的評分是否達到 Q-對齊：即其能否根據強參考策略的 Q 值來對動作進行排序。這使我們能在任何訓練運算之前比較信號，並將信號品質與其他工程選擇區分開來。我們將 QVal 實現為 QVal-v1.0，針對 21 種密集監督方法，橫跨四個不同環境與七個方法論家族，並基於六個開放權重模型主幹進行超過 1,200 項評估實驗。我們發現，簡單的提示基準始終優於文獻中近期提出的密集監督方法，且效能在家族內部呈現強烈聚集。這些發現在不同模型規模、環境與觀察模態下均保持一致。QVal 旨在易於擴展至新環境與新方法，使研究人員能在任何訓練運算之前對密集監督方法進行迭代改進。

English

LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions. Dense supervision methods aim to solve this problem by scoring intermediate steps, from intrinsic confidence to self-distillation and embedding similarities. However, it is common practice to evaluate them by measuring the downstream performance of a training pipeline that integrates them. This is expensive, conflates supervision quality with training engineering confounders, and renders different methodological families requiring distinct training setups incomparable. As a result, dense supervision methods are rarely benchmarked on common ground. We introduce QVal, a training-free testbed for directly evaluating dense supervision signals. Given a state-action pair, QVal measures how well a method's score is Q-aligned: whether it orders actions according to the Q-values of a strong reference-policy. This lets us compare signals before any training run and separate signal quality from other engineering choices. We instantiate QVal as QVal-v1.0, benchmarking 21 dense supervision methods across four diverse environments and seven methodological families, with over 1.2K evaluation experiments across six open-weight model backbones. We find that simple prompting baselines consistently outperform recent dense supervision methods from the literature, and that performance clusters strongly by family. These findings hold across model sizes, environments, and observation modalities. QVal is designed to be easily extensible to new environments and methods, enabling researchers to iterate on dense supervision methods before any training run.