流形老虎機：基於大型語言模型潛在幾何結構的貝氏課程學習

摘要

強化學習（RL）是提升大型語言模型（LLM）推理能力的核心方法，其訓練效率關鍵取決於最佳化過程中問題的取樣方式。現有的自適應課程學習方法通常優先選取中等難度的提示詞，將問題選擇視為標準的多臂老虎機問題（各臂獨立），卻忽略了任務空間本身具有結構性與異質性的本質。在本研究中，我們將問題取樣框架化為具備內生非平穩性的流形結構多臂老虎機問題：問題之間透過模型的潛在表徵空間相互關聯，而取樣決策會引導學習訊號在該空間中的演化方向。為落實此觀點，我們提出貝氏流形課程（BMC）——一個具結構感知的框架，將問題組織成階層式任務樹，並運用貝氏學習來引導取樣。實驗結果顯示，不同的取樣策略會在生產力（學習訊號）、多樣性（任務流形覆蓋率）與效用（評估相關性）之間產生非平凡的取捨。這些結果表明，僅優先考慮難度並不足以在最終下游表現上取得優異成果，凸顯了在問題取樣中納入結構性與類型感知的重要性。

English

Reinforcement learning (RL) is a central approach for improving reasoning capabilities in large language models (LLMs), where training efficiency depends critically on how problems are sampled during optimization. Existing adaptive curriculum learning methods typically prioritize prompts of intermediate difficulty, treating problem selection as a standard bandit problem with independent arms and overlooking the structured, heterogeneous nature of the task space. In this work, we frame problem sampling as a manifold-structured bandit problem with endogenous non-stationarity: problems are related through the model's latent representation space, and sampling decisions can steer how learning signals evolve across that space. To operationalize this perspective, we introduce Bayesian Manifold Curriculum (BMC), a structure-aware framework that organizes problems into a hierarchical task tree and applies Bayesian learning to guide sampling. Empirically, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance). These results show that prioritizing difficulty alone is insufficient for strong downstream performance, highlighting the importance of incorporating structure and type-awareness into problem sampling.