代理探索與可重用指導：通過代理引導的更新信號實現模塊化大語言模型後訓練範式

摘要

後訓練對於提升大型語言模型（LLMs）的領域專用能力至關重要，然而現有的獎勵優化與分佈匹配方法將策略探索與分佈對齊緊密耦合。這種耦合迫使模型直接在策略模型上進行昂貴的探索，並嚴重阻礙了優化信號的非同步生成、重複使用以及跨模型遷移。本文提出了一種新穎的後訓練框架——代理引導更新信號傳遞（Proxy-guided Update Signal Transfer, PUST），從根本上將更新信號探索與分佈對齊解耦。PUST 不直接使用主模型進行高成本探索，而是採用輕量級代理模型作為高效測試平台，以發現高獎勵行為。我們提取代理模型初始狀態與優化後狀態之間的相對改進信號，將這種方向性更新傳遞給主模型，引導其策略對齊。這種由代理探索、更新信號提取與信號傳遞組成的解耦流程，顯著降低了計算開銷，並使優化信號能夠非同步生成、緩存及重用。關鍵在於，通過傳遞相對改進而非絕對策略分佈，PUST 自然支持從弱到強的改進以及無縫的跨模型遷移。在 Qwen3 系列模型上針對數學和程式碼領域的系統評估表明，從明顯較弱的代理模型中提取的更新信號，能夠穩健且可調控地增強較強的主模型。最終，PUST 將後訓練從單一性的在線優化過程轉變為高度模組化、可重用且具成本效益的新範式。

English

Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.