RynnValue：以時間距離擴展機器人價值基礎模型

摘要

通用獎勵模型日益成為擴展機器人學習的瓶頸，然而從大規模異構語料中學習價值相關能力的方法仍未得到充分探索。現有方法將監督與任務內部錨點綁定，例如偏好或歸一化進度，而這些錨點皆無法在不同具身形態與資料來源之間乾淨地遷移。我們提出 RynnValue，一個開源的機器人操作價值基礎模型，它以時間距離取代這些錨點——即從觀察到語言指定目標之間的定向剩餘成本。由於時間距離標籤可直接從時間戳中推導，RynnValue 無需偏好或進度標註即可擴展至超過 7,000 小時、約 300 萬條指令條件化片段。為了使時間價值學習在大規模下保持可靠，我們結合隨機時間採樣、時間順序洗牌與價值隔離注意力，以抑制使預測對失敗和倒退不敏感的捷徑。RynnValue 在無偏好標籤的訓練下，於 RBM-EVAL-OOD 取得 0.675 的平均肯德爾 tau_a 係數，超越完全偏好監督的當前最佳方法（0.655），且是僅用進度監督的對照模型（0.292）的兩倍以上，同時能零樣本泛化至未見過的任務、具身形態與視角。透過基於勢能的塑形轉換為稠密獎勵後，它將真實世界策略成功率從 52.5% 提升至 72.5%（在線），並從 63.8% 提升至 82.5%（離線）。這些結果確立了時間距離作為可擴展監督目標及通用機器人策略實用獎勵介面的地位。

English

General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.