WarpSAC：通过重新思考探索与利用迈向可扩展离线策略强化学习的巅峰
WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
August 25, 2026
作者: Zihao Wu, Hongyao Tang, Yi Ma, Huizhong Song, Pengyi Li, Yifu Yuan, Fei Ni, Jinyi Liu, Wei Wei, Jianrong Wang, Yan Zheng, Jianye Hao
cs.AI
摘要
大规模并行仿真改变了离策略强化学习（RL）的训练数据体制，这给为数据受限回放设计的稳定器带来了挑战。通过跨越八个基准家族的受控实验，我们证明这些稳定器具有数据体制依赖性：参数归一化在回放覆盖范围较窄时有所帮助，但在数据充裕时限制了价值拟合；而裁剪双Q在高吞吐操作中可以放宽。年龄偏置回放加权在不同体制下均能提升学习效率，尤其在网络容量有限时效果显著。
基于这些发现，我们提出了WarpSAC——一个具备体制感知能力的离策略RL算法家族。WarpSAC采用样本权重衰减（Sample Weight Decay）实现高效利用，并提供两种变体：面向数据受限CPU规模训练的WarpSAC-L（归一化开启，裁剪双Q）和面向数据充裕GPU并行训练的WarpSAC-A（归一化关闭，单Q）。WarpSAC在九个CPU规模环境中将归一化得分-步数AUC较FlashSAC提升了4.5%，在十四个GPU并行环境中提升了23.1%。它将UnitreeG1TransportBox-v1的成功率从19.8%提升至96.4%，在MuJoCo Playground上将平均归一化墙钟时间AUC提升了19.1%，并在Unitree G1上实现了比FlashSAC快36.4%的仿真到现实部署。这些结果表明，可扩展的离策略RL应根据可用的数据体制调整其稳定器。
English
Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity.
Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.