QVal：低成本评估长时域LLM智能体的密集监督信号

摘要

大型语言模型代理越来越多地在长程任务中运行，单一轨迹可能包含数百甚至数千个动作。在这些场景中，仅结果奖励提供的指导过于稀疏，无法告知模型中间动作的质量。密集监督方法试图通过评分中间步骤来解决这一问题，方法涵盖内在置信度、自蒸馏以及嵌入相似度等。然而，常见的评估方式是通过测量整合这些方法的训练管线的下游性能。这种方法成本高昂，会将监督质量与训练工程中的混淆因素混为一谈，并导致需要不同训练配置的方法家族之间无法进行比较。因此，密集监督方法很少在共同基准上进行评估。我们提出QVal，这是一个无需训练的测试平台，用于直接评估密集监督信号。给定一个状态-动作对，QVal衡量方法的评分是否与Q值对齐：即是否根据强参考策略的Q值对动作进行排序。这使我们能在任何训练运行前比较信号，并将信号质量与其他工程选择分离开。我们实例化了QVal-v1.0，在四个不同环境和七个方法家族中对21种密集监督方法进行了基准测试，跨六个开放权重模型骨干进行了超过1,200次评估实验。我们发现，简单的提示基线方法始终优于文献中提出的最新密集监督方法，且性能按方法家族显著聚类。这些发现在不同模型规模、环境和观测模态下均成立。QVal被设计为易于扩展至新环境和新方法，使研究人员能够在任何训练运行之前迭代密集监督方法。

English

LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions. Dense supervision methods aim to solve this problem by scoring intermediate steps, from intrinsic confidence to self-distillation and embedding similarities. However, it is common practice to evaluate them by measuring the downstream performance of a training pipeline that integrates them. This is expensive, conflates supervision quality with training engineering confounders, and renders different methodological families requiring distinct training setups incomparable. As a result, dense supervision methods are rarely benchmarked on common ground. We introduce QVal, a training-free testbed for directly evaluating dense supervision signals. Given a state-action pair, QVal measures how well a method's score is Q-aligned: whether it orders actions according to the Q-values of a strong reference-policy. This lets us compare signals before any training run and separate signal quality from other engineering choices. We instantiate QVal as QVal-v1.0, benchmarking 21 dense supervision methods across four diverse environments and seven methodological families, with over 1.2K evaluation experiments across six open-weight model backbones. We find that simple prompting baselines consistently outperform recent dense supervision methods from the literature, and that performance clusters strongly by family. These findings hold across model sizes, environments, and observation modalities. QVal is designed to be easily extensible to new environments and methods, enabling researchers to iterate on dense supervision methods before any training run.