先排序後行動：基於幀序進展的無獎勵控制

摘要

我們提出 Rank-Then-Act (RTA) 框架，這是一種無需環境獎勵、僅從專家影片示範學習控制策略的方法。RTA 離線訓練一個視覺語言模型 (VLM)，使其成為基於進展的順序評分器，並使用群組相對策略最佳化 (GRPO) 目標對打亂的幀序列進行訓練，迫使模型從視覺語義而非時間線索中恢復時間順序。關鍵在於，我們不直接將評分器用作標量獎勵模型，而是提出一種基於相關性的獎勵函數，用於強化學習：在每個互動視窗中，我們計算預測進展排名與真實時間索引之間的斯皮爾曼等級相關，從而產生一個有界、尺度不變的學習訊號。這種設計將獎勵學習與絕對校準脫鉤，並能穩定地在不同任務與環境之間遷移。我們在離散控制基準（PyBoy：Catrap、Kirby）與連續控制任務（PointMaze、MetaWorld）上評估 RTA。RTA 始終達到或超越先前的基於影片的獎勵學習方法與基於排名的基準，同時展現單一預訓練進展評分器在跨任務重複使用上的強勁能力。我們的結果表明，基於影片衍生的順序訊號的相關結構化監督足以進行策略學習，為明確的獎勵設計提供了一種可擴展的替代方案。

English

We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains a Vision-Language Model (VLM) offline as a progress-based ordinal scorer, using a Group Relative Policy Optimization (GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function for reinforcement learning: at each interaction window, we compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms prior video-based reward learning methods and rank-based baselines, while demonstrating strong cross-task reuse of a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design.