懂得何时不重用：自主LLM后训练中的条件化经验迁移

摘要

大语言模型具有广泛的能力，但将其适配到不断演化的领域、工具和需求通常需要反复的后训练。自主系统通过提出更新、训练候选模型并利用评估反馈来选择后续提案，从而将这一过程中的部分环节自动化。随着证据的不断积累，一个核心问题浮现出来：当后续训练已经改变了父模型之后，哪些过去的更新证据仍然可行？更新的效果取决于其父模型、数据以及训练阶段。将过去的成功视为与上下文无关的许可，可能会浪费算力；若由此产生的子模型被采纳，还可能使后续的训练轨迹退化。我们将这一问题形式化为条件经验迁移，并引入边界校准干预迁移（Boundary-Calibrated Intervention Transfer, BCIT）——一种在改变权重的训练之前对经验复用进行授权的方法。BCIT 将观察到的效果绑定到其来源上下文，检查适用条件，否决存在明确硬冲突的候选，并在需要时通过有界训练试验获取当前状态的证据。完全训练完成的候选仍然要遵循统一的采纳规则，而且只有被观察到的事件才能扩展记忆。在一个经过金融推理、文本到SQL和函数调用适配的4B参数模型上，候选更新在不同评估场景中表现出异质的目标效应与保留效应。在候选、证据和算力均匹配的条件下，与所评估的备选方法相比，BCIT 授权更少的有害更新，并在同等预算下获得更高的最终模型质量。这些结果支持将经验授权视为自主后训练中的一个独立问题。

English

Large language models offer broad capabilities, but adapting them to evolving domains, tools, and requirements often entails repeated post-training. Autonomous systems automate parts of this process by proposing updates, training candidates, and using evaluation feedback to select subsequent proposals. As evidence accumulates, a central problem emerges: which past update evidence remains actionable after subsequent training has changed the parent model? An update's effect depends on its parent, data, and training stage. Treating past success as context-free permission can waste compute. If the resulting child is promoted, it can also degrade the subsequent training trajectory. We formulate this problem as conditional experience transfer and introduce Boundary-Calibrated Intervention Transfer (BCIT), a method that authorizes experience reuse before weight-changing training. BCIT binds an observed effect to its source context, checks applicability conditions, vetoes candidates with named hard conflicts, and obtains current-state evidence through a bounded training trial when needed. Fully trained candidates still face a shared adoption rule, and only observed events extend memory. On one 4B model adapted across finance reasoning, text-to-SQL, and function calling, candidate updates exhibit heterogeneous target and retention effects across the evaluated contexts. Under matched candidates, evidence, and compute, BCIT authorizes fewer harmful updates and attains higher equal-budget final-model quality than the evaluated alternatives. These results support treating experience authorization as a distinct problem in autonomous post-training.

懂得何时不重用：自主LLM后训练中的条件化经验迁移

Knowing When Not to Reuse: Conditional Experience Transfer in Autonomous LLM Post-Training

摘要

Support