用於擴展RLVR中推理覆蓋範圍的難度自適應樹狀結構策略優化

摘要

可驗證獎勵強化學習（RLVR）一直是近期大型推理模型成功的核心。然而，儘管 RLVR 顯著提升單樣本準確率，卻常因訓練期間探索有限，而未能擴展模型固有的推理覆蓋率（pass@k）。為解決此問題，我們最佳化訓練時 rollout 的結構設計，以提升 pass@k。我們的分析指出三項關鍵設計原則：(1) 難度自適應 rollout 除了可作為效率啟發式之外，還能在擴展 pass@k 上扮演重要角色；(2) 樹狀 rollout 在發現正確答案方面優於平行取樣；(3) 句子熵引導的分岔克服了 token 層級分支的局部化現象，以最大化語意多樣性。基於這些洞見，我們提出 DATPO（Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization，難度自適應句子熵引導樹狀策略優化）。DATPO 將難度自適應樹狀搜尋與同層兄弟多樣性優勢項整合，明確促進語意多樣性，以在訓練期間擴展推理覆蓋率。在數學推理基準上的實驗顯示，DATPO 優於基線，尤其在 pass@k 方面更為突出，這直接轉化為更優異的測試時擴展效能。

English

Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.