用於程式碼最佳化的強化學習

摘要

用於程式碼正確性的強化學習現在已確立：讓模型生成程式，對照隱藏測試案例執行，並獎勵通過的解。將此概念擴展至程式碼最佳化看似直接：只需將執行時間加入獎勵。然而在實務中，一旦時間驅動獎勵，測量雜訊、獎勵稀疏或 GRPO 不穩定性等小問題便會壓過訊號，使強化學習失敗：生成程式碼幾乎沒變快，且更多程式會失敗。我們透過三個階段讓執行時間成為可學習的目標：(1) 程式碼的測試方式——建構 DMC-Optim，搭配大型最佳化測試與校準後的沙盒；(2) 速度如何轉化為獎勵——在強化學習環境中組合正確性與速度，並使用離線模擬器預測最有潛力的配置；(3) 模型從該獎勵中學習的方式——調整 GRPO 與評估方法，以適應更稀疏、更具雜訊的計時執行環境。在 DMC-Optim 上，最佳的最佳化感知配置將 Qwen 2.5 7B 的嚴格前 50% pass@1 從 18.0% 提升至 31.3%，CWM 32B 則從 30.7% 提升至 50.4%。這些提升在更嚴格的百分位（如前 30%）進一步擴大，CWM 32B 的相對進步達 125%，同時保留了純正確性分數。當計時沙盒品質下降時，穩健的最佳化強化學習依評估標準不同，較標準 RLVR 提升了 100% 至 200%。在 LCB 上，CWM 32B 在基於中位數樣本的速度比較中，對上標準 RLVR 最高可贏得 83%。相較於每位問題最快的正確人類提交，其複雜度類別提升率約為人類的一半（14% 對 28%）。

English

RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).