WarpSAC：透過重新思考探索與利用邁向可擴展離策略強化學習的巔峰
WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
August 25, 2026
作者: Zihao Wu, Hongyao Tang, Yi Ma, Huizhong Song, Pengyi Li, Yifu Yuan, Fei Ni, Jinyi Liu, Wei Wei, Jianrong Wang, Yan Zheng, Jianye Hao
cs.AI
摘要
大規模並行模擬改變了離策略強化學習（RL）的訓練數據體制，對為數據有限重放設計的穩定器構成挑戰。透過八個基準測試系列的對照實驗，我們顯示這些穩定器依賴於數據體制：參數正規化有助於狹窄的重放覆蓋，但在數據豐富時限制了價值擬合；而裁剪雙Q在高吞吐量的操作任務中可以放寬。年齡偏置的重放加權在各個體制下均能提升學習效率，尤其是在網絡容量有限時。基於這些發現，我們提出了 WarpSAC，一個數據體制感知的離策略 RL 算法家族。WarpSAC 使用樣本權重衰減來實現高效利用，並提供兩個變體：WarpSAC-L（參數正規化開啟，裁剪雙Q）用於數據有限的 CPU 規模訓練，以及 WarpSAC-A（參數正規化關閉，單Q）用於數據豐富的 GPU 並行訓練。與 FlashSAC 相比，WarpSAC 在九個 CPU 規模環境中將標準化分數-步數 AUC 提升了 4.5%，在十四個 GPU 並行環境中提升了 23.1%。它將 UnitreeG1TransportBox-v1 的成功率從 19.8% 提高到 96.4%，在 MuJoCo Playground 上將平均標準化牆鐘時間 AUC 提升了 19.1%，並實現了比 FlashSAC 快 36.4% 的 Unitree G1 模擬到現實部署。這些結果表明，可擴展的離策略 RL 應根據可用的數據體制調整其穩定器。
English
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity.
Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.