過時但穩定：用於穩定非同步強化學習的過時適應性信任區域

摘要

非同步強化學習透過將軌跡生成與最佳化過程解耦以提升吞吐量，但陳舊性（staleness）是不可避免的副產物，其成因包括策略延遲、引擎延遲及專家混合路由。從信賴區域（trust-region）的觀點來看，此失配問題至關重要：訓練-推論分歧決定了有限時域界限中的近似誤差，而PPO裁切（PPO clipping）僅能限制抽樣的向外更新，作為一種抽樣替代而非全局策略約束。因此，在陳舊軌跡影響最顯著的非同步機制下，高陳舊性更新依然缺乏有效控制。我們提出適應性陳舊度信賴區域（Staleness-Adaptive Trust Region, SAT），該方法利用分離的抽樣對數比作為實用的陳舊度代理，透過基於陳舊度的核縮放識別每一批次內的高失配尾部，並僅收縮名義PPO區間中符號選取的端點。此舉在普通詞元上保留基準行為，同時對新攔截的向外帶實施更保守的更新。我們證明了相較於PPO的局部區間包含性與逐點悲觀性，展示了適應性規則如何在異質陳舊度下重塑更新幾何結構。我們在基於Qwen3-30B-A3B-Base建構的解耦非同步RL設定中評估SAT，採用SGLang作為推理引擎、Megatron進行訓練。在此設定下，SAT-GSPO搭配R3達到了觀測到的最佳AIME24平均@8，在滯後1為35.83、滯後8為34.79；而SAT-GSPO在滯後1達到34.17。適應性裁切與路由重播分別針對失配尾部與路由不一致性，作為互補的穩定器。總體而言，使裁切區間與陳舊度異質性對齊，能有效穩定非同步強化學習。

English

Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: training-inference divergence governs approximation error in finite-horizon bounds, whereas PPO clipping only gates sampled outward updates, acting as a sampled surrogate rather than a full-policy constraint. As a result, high-staleness updates remain weakly controlled in the asynchronous regime where stale rollouts matter most. We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy, identifies high-mismatch tails within each batch via staleness-based kernel scaling, and contracts only the sign-selected endpoint of the nominal PPO interval. This preserves baseline behavior on ordinary tokens while enforcing more conservative updates on newly intercepted outward bands. We prove local interval containment and pointwise pessimism relative to PPO, showing how the adaptive rule reshapes update geometry under heterogeneous staleness. We evaluate SAT in a decoupled asynchronous RL setup built on Qwen3-30B-A3B-Base, using SGLang as the inference engine and Megatron for training. In this setting, SAT-GSPO w/ R3 achieves the best observed AIME24 avg@8, reaching 35.83 at lag 1 and 34.79 at lag 8, while SAT-GSPO reaches 34.17 at lag 1. Adaptive clipping and routing replay act as complementary stabilizers targeting mismatch tails and routing inconsistency, respectively. Overall, aligning clip intervals with staleness heterogeneity effectively stabilizes asynchronous RL.