机器人学习中的进展奖励建模：一项全面综述

摘要

机器人学习发生在动态环境和大行为空间中。终端成功信号仅告知机器人任务是否完成，却无法揭示当前行为是在推进进展、停滞不前还是造成倒退。为此，近年来越来越多的研究探索在任务执行过程中提供反馈的进度奖励机制。然而，当前文献缺乏统一框架：现有方法在观测数据、目标设定、输出信号、监督来源和评估协议上存在差异，导致难以进行方法比较并理解其验证结果的实际意义。本综述为机器人学习的进度奖励建模提供统一视角，将领域组织为三个关联环节。首先研究进度模型的接口——通过考察模型接收的信息及其产生的进度信号形式，从外部定义问题边界。继而深入模型内部，分析构建进度信号的方法，揭示进度估计与奖励生成背后的不同假设与机制。最后检验支撑这些方法的数据与基准，阐明进度监督的获取方式及不同评估的实际测量内容。这三个视角共同串联起进度模型的本质定义、构建方式与质量验证。我们进一步总结了现有方法的主要局限，并探讨了未来研究方向。

English

Robotic learning takes place in dynamic environments with large behavior spaces. A terminal success signal only tells the robot whether the task is completed. It does not explain whether the current behavior is making progress, remaining unchanged, or undoing earlier progress. For this reason, recent studies have increasingly explored progress rewards that provide feedback during task execution. However, the current literature lacks a shared framework. Existing methods use different observations, goal specifications, output signals, supervision sources, and evaluation protocols. This makes it difficult to compare them and understand what their results actually validate. In this survey, we provide a unified view of progress reward modeling for robotic learning. We organize the field in three connected steps. We first study the interface of a progress model. This defines the problem from the outside by asking what information the model receives and what form of progress signal it produces. We then move inside the model and study the methods used to construct this signal. This reveals the different assumptions and mechanisms behind progress estimation and reward generation. Finally, we examine the data and benchmarks that support these methods. This shows how progress supervision is obtained and what different evaluations actually measure. Together, these three perspectives connect what a progress model is, how it is built, and how its quality is validated. We further summarize the main limitations of current approaches and discuss future research directions.