機器人學習中的進度獎勵建模：一項全面綜述

摘要

機械人學習發生在具有大規模行為空間的動態環境中。最終的成功訊號僅能告訴機械人任務是否完成，卻無法說明當前行為是否正在取得進展、保持不變，或是在抵消先前的進展。為此，近期的研究越來越重視在任務執行過程中提供反饋的進度獎勵。然而，當前文獻缺乏一個共同的框架。現有方法使用了不同的觀測量測、目標規格、輸出訊號、監督來源與評估協議。這使得難以對它們進行比較，也難以理解其實際驗證了什麼結果。在本篇綜述中，我們針對機械人學習中的進度獎勵建模提出一個統一的觀點。我們將此領域組織為三個相互關聯的步驟。首先，我們研究進度模型的外部介面，透過探討模型接收什麼資訊以及產生何種形式的進度訊號，從外部定義問題。接著，我們深入模型內部，研究構建此訊號所用的方法。這揭示了進度估計與獎勵生成背後不同的假設與機制。最後，我們檢視支持這些方法的資料與基準測試。這說明了進度監督是如何獲取的，以及不同的評估實際衡量了什麼。這三個觀點共同連結了進度模型是什麼、如何構建以及其品質如何驗證。我們進一步總結當前方法的主要限制，並討論未來的研究方向。

English

Robotic learning takes place in dynamic environments with large behavior spaces. A terminal success signal only tells the robot whether the task is completed. It does not explain whether the current behavior is making progress, remaining unchanged, or undoing earlier progress. For this reason, recent studies have increasingly explored progress rewards that provide feedback during task execution. However, the current literature lacks a shared framework. Existing methods use different observations, goal specifications, output signals, supervision sources, and evaluation protocols. This makes it difficult to compare them and understand what their results actually validate. In this survey, we provide a unified view of progress reward modeling for robotic learning. We organize the field in three connected steps. We first study the interface of a progress model. This defines the problem from the outside by asking what information the model receives and what form of progress signal it produces. We then move inside the model and study the methods used to construct this signal. This reveals the different assumptions and mechanisms behind progress estimation and reward generation. Finally, we examine the data and benchmarks that support these methods. This shows how progress supervision is obtained and what different evaluations actually measure. Together, these three perspectives connect what a progress model is, how it is built, and how its quality is validated. We further summarize the main limitations of current approaches and discuss future research directions.