通用推理的可遷移性：多領域RLVR的自動化課程

摘要

具備可驗證獎勵的強化學習已從單一領域訓練擴展至涵蓋數學、程式設計與科學的多領域推理任務組合。然而，訓練課程（各領域的採樣頻率）通常固定不變或手動調整，儘管推理技能在領域間的轉移並不均勻。現有的基於可學習性的課程設計雖能適應策略當前進步的領域，但無法判斷選定領域的一次梯度更新是否有利於其他領域。本文提出「轉移感知課程」（Transfer-Aware Curriculum, TAC），一種賭臂式的在線課程機制，優先選取那些更新能廣泛惠及訓練組合中其餘領域的領域。TAC 重複利用強化學習訓練過程中已產生的訊號：各領域的優勢函數反映局部可學習性，而從正在計算的 GRPO 步驟中提取的投影梯度，則透過梯度幾何對齊來估計跨領域的可轉移性，且計算成本極低（額外耗時不到 1%）。在包含六個領域的推理任務組合上，TAC 在 Qwen3-1.7B 與 Llama3.2-3B 模型上均達到最佳宏觀平均準確率，勝過比例隨機採樣、手動設計的排程以及僅基於可學習性的賭臂式方法，並相較後者最高提升 2.8 個百分點（相對提升 10%）。消融實驗顯示，移除可轉移性項後效能急遽下降，且在不平衡的訓練混合中，TAC 仍保持穩健，而僅基於可學習性的課程則會過度投入佔優領域。我們的研究確立了跨領域可轉移性作為多領域 RLVR 課程設計的一項關鍵訊號。

English

Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost (<1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR.