先排序后行动：基于帧序进展的无奖励控制

摘要

我们提出了“先排序后行动”（Rank-Then-Act, RTA）框架，用于在无环境奖励的情况下，从专家视频演示中学习控制策略。RTA 离线训练一个视觉语言模型（VLM），使其成为基于进度的顺序评分器。该训练采用组相对策略优化（GRPO）目标，在打乱的帧序列上进行，迫使模型仅依据视觉语义而非简单的时间线索恢复时间顺序。关键在于，我们不直接将该评分器用作标量奖励模型，而是提出一种基于相关性的奖励函数用于强化学习：在每个交互窗口中，我们计算预测进度排名与真实时间索引之间的斯皮尔曼秩相关系数，从而得到一个有界且尺度不变的 learning 信号。这种设计将奖励学习与绝对校准解耦，使得单个预训练进度评分器能够稳定地迁移到不同任务和环境。我们在离散控制基准（PyBoy：Catrap、Kirby）和连续控制任务（PointMaze、MetaWorld）上评估了 RTA。RTA 始终能够匹配或超越以往基于视频的奖励学习方法以及基于排名的基线方法，同时展现出单个预训练进度评分器在跨任务复用方面的强大能力。我们的结果表明，基于视频序数信号的关联结构监督足以支持策略学习，为显式奖励设计提供了一种可扩展的替代方案。

English

We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains a Vision-Language Model (VLM) offline as a progress-based ordinal scorer, using a Group Relative Policy Optimization (GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function for reinforcement learning: at each interaction window, we compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms prior video-based reward learning methods and rank-based baselines, while demonstrating strong cross-task reuse of a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design.