ISO：一個 RLVR 原生最佳化堆疊

摘要

基於可驗證獎勵的強化學習（RLVR）正迅速推進語言模型的推理能力，然而，將獎勵反饋轉化為權重空間更新的優化層卻仍未被充分理解。基於我們先前的研究（Zhu 等人，2025），我們透過模型權重的奇異結構來探討這個缺失的層，並發現了譜繼承現象：RLVR 能夠重複使用基礎模型的權重譜，同時透過相關的輸入與輸出奇異幀的變化來獲得新的行為。我們將譜繼承具體實現為等譜優化（ISO），這是一個原生於 RLVR 的固定譜優化框架，並提供互補的離線與線上實例。離線版本 ISO-Merger 將共享基礎模型之專家成員的幀變化合併為單一固定譜模型，無需任何合併後的數據、交互取樣、梯度更新或在策略蒸餾（OPD）。它恢復了互補的專家能力，並在比較的無數據合併方法中達到了最強的綜合性能。線上版本 ISO-Optimizer 將選定的基礎優化器（包括 AdamW 與 Muon）應用於幀變量，同時保持基礎譜固定。在參數規模從 1.5B 到 8B 的推理與程式碼任務中，ISO-Optimizer 在報告的運行中提升了準確率，並以顯著更少的訓練步數達到匹配的分數。以 Qwen3-8B-Base 為例，AdamW 在 270 個訓練步數後達到 0.495 的聚合準確率，而 ISO-AdamW 僅需 100 個訓練步數便達到相同準確率，並在 210 個訓練步數後進一步提升至 0.509。綜上所述，ISO 為 RLVR 中缺失的優化層提供了具體解答：與其全盤繼承預訓練的優化方式，不如在獎勵驅動的適應結構基礎上設計後訓練——繼承譜，優化幀。

English

Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through the singular structure of model weights and identify spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior through changes in the associated input and output singular frames. We operationalize spectral inheritance as Isospectral Optimization (ISO), an RLVR-native, fixed-spectrum optimization framework with complementary offline and online instantiations. Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model, requiring no post-merge data, rollouts, gradient updates, or on-policy distillation (OPD). It recovers complementary specialist capabilities and achieves the strongest aggregate performance among the compared data-free merging methods. Online, ISO-Optimizer applies a chosen base optimizer, including AdamW and Muon, to the frame variables while keeping the base spectra fixed. Across reasoning and coding tasks ranging from 1.5B to 8B parameters, ISO-Optimizer improves accuracy in the reported runs and reaches matched scores with substantially fewer training steps. On Qwen3-8B-Base, AdamW reaches an aggregate accuracy of 0.495 after 270 training steps. ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 after 210 training steps. Together, ISO offers a concrete answer to RLVR's missing optimization layer: rather than inheriting pre-training optimization wholesale, design post-training around the structure of reward-driven adaptation: inherit the spectrum, optimize the frames.