面向代码优化的强化学习

摘要

面向代码正确性的强化学习已是成熟技术：让模型生成程序，将其与隐藏测试用例对照运行，并对通过测试的方案给予奖励。将其扩展至代码优化看似直接：只需将执行时间加入奖励函数。但在实践中，一旦执行时间驱动奖励，测量噪声、奖励稀疏性或GRPO不稳定等小问题便会淹没有效信号，导致强化学习失效：生成的方案速度提升甚微，且更多方案可能无法通过测试。我们通过三个阶段使执行时间成为可学习的要素：(1) 测试代码的方式——构建包含大规模优化测试与校准沙箱的DMC-Optim数据集；(2) 将执行速度转化为奖励的方式——在强化学习环境中组合正确性与速度，并利用离线模拟器预测最具潜力的配置；(3) 模型从该奖励中学习的方式——针对更稀疏、噪声更大的定时执行场景调整GRPO与评估方法。在DMC-Optim数据集上，最强优化感知配置将严格前50%的pass@1从Qwen 2.5 7B的18.0%提升至31.3%，CWM 32B则从30.7%提升至50.4%。这些增益在更严格的百分位（如前30%）进一步提升，CWM 32B实现125%的相对提升，同时保持纯正确性分数不变。当定时沙箱降级时，稳健优化强化学习相比标准RLVR在不同评估标准下获得100%至200%的改进。在LCB基准上，CWM 32B在样本中位数速度对比中最高胜出标准RLVR达83%。相较于每个问题的正确人类最快提交，其复杂度等级改进率约为人类水平的一半（14%对28%）。

English

RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-50% pass@1 from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. These gains further increase at stricter percentiles such as top-30%, with 125% relative improvement for CWM 32B, while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL reaches 100% to 200% improvement over standard RLVR, depending on the evaluation criterion. On LCB, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).