以函数感知的中间填充作为编码智能体基础模型的中期训练

摘要

编码代理必须将外部工具返回结果整合到持续推理中——这一能力是标准从左到右代码预训练仅以其前向方向所暴露出来的。我们观察到，编码代理的动作-观察-持续循环在结构上同构于函数调用点，其中调用者绑定参数，被调用者返回在其他地方计算的值，而下游代码消费该值。这种条件化结构在普通代码中以互联网规模存在。我们通过函数感知的填充中间（FIM）中训练来利用这一点：这是一种自监督目标，它掩码通过程序依赖图分析以及复杂度-可推断性双标准所选择的函数。我们在一个来自968个GitHub仓库、经过去污染处理的26亿token语料上对Qwen2.5-Coder-Instruct（7B/14B）和Qwen3-8B进行中训练，然后应用现有的代理后训练流程。中训练在SWE-Bench-Verified上提升+2.8/+3.0（7B/14B），在Qwen3-8B上提升+3.2；在相同模型上SWE-Bench-Lite的提升分别为+3.7/+4.0/+5.4。该改进在两种后训练流程（R2E-Gym，SWE-Smith）以及非Qwen2.5基座（使用SWE-Lego的Qwen3-8B）上均保持一致。除了领域内增益，中训练还缓解了代理后训练否则会在非代理编码（如LiveCodeBench）和非编码工具使用基准（tau-bench，BFCL）上造成的能力侵蚀：尽管中训练语料仅包含Python代码，但函数调用归纳偏置在后训练中得以保留并产生一致增益。

English

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.