面向通用推理的可迁移性：多领域RLVR的自动课程

摘要

将基于可验证奖励的强化学习（RLVR）从单领域训练扩展至涵盖数学、编程和科学的多领域推理任务套件已成为趋势。然而，训练课程（即各领域采样频率）通常固定或手动调整，尽管推理技能在不同领域间的迁移程度并不均衡。现有基于可学习性的课程策略会适应策略当前改进的方向，但忽略了所选领域上的梯度更新是否有利于其余领域。本文提出迁移感知课程（TAC），一种赌博机式在线课程方法，优先选择更新收益能广泛惠及训练套件中其他领域的领域。TAC重用了RL训练中已产生的信号：各领域的优势值表征局部可学习性，而从正在计算的GRPO步骤中提取的投影梯度，通过梯度几何对齐以可忽略的成本（<1%的挂钟时间开销）估计跨领域迁移性。在一个包含六个领域的推理套件上，TAC在Qwen3-1.7B和Llama3.2-3B模型上均取得了最佳宏观平均准确率，优于比例随机采样、手工设计调度方案以及纯可学习性赌博机方法，相较后者最高提升2.8个百分点（相对提升10%）。消融实验表明，去除迁移性项后性能急剧下降，且在不平衡训练混合场景中，当纯可学习性课程过度聚焦于主导领域时，TAC仍保持稳健。我们的研究确立了跨领域迁移性作为多领域RLVR课程设计的关键信号。

English

Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost (<1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR.