代理探索与可复用引导：一种通过代理引导更新信号的模块化大语言模型后训练范式

摘要

后训练对提升大语言模型（LLMs）的领域特定能力至关重要，但现有的奖励优化和分布匹配方法将策略探索与分布对齐紧密耦合。这种耦合迫使模型直接在策略模型上进行昂贵的探索，严重阻碍了优化信号的异步生成、复用以及跨模型迁移。在本文中，我们提出代理引导的更新信号迁移（PUST），一种新型后训练框架，从根本上将更新信号的探索与分布对齐解耦。PUST 并非使用主模型进行昂贵的探索，而是采用轻量级代理模型作为高效的试验平台，用于发现高奖励行为。我们提取代理模型在初始状态与优化状态之间的相对改进信号，并将这一方向性更新迁移至主模型，以指导其策略对齐。这一解耦流程——包含代理探索、更新信号提取与信号迁移——显著降低了计算开销，并使优化信号能够异步生成、缓存和复用。关键在于，通过迁移相对改进而非绝对策略分布，PUST 天然支持从弱到强的改进以及无缝的跨模型迁移。在 Qwen3 系列模型上针对数学和代码领域的系统评估表明，从显著更弱的代理模型中提取的更新信号，能够稳健且可调节地增强更强的主模型。最终，PUST 将后训练从单一的在线优化过程转变为高度模块化、可复用且成本高效的范式。

English

Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.