以函式感知的填充中間任務作為程式碼代理基礎模型的中期訓練

摘要

編碼代理必須將外部工具回傳結果整合到持續推理中——這項能力是標準的從左到右程式碼預訓練僅在其前向方向上所展現的。我們觀察到，編碼代理的行動-觀察-延續迴圈在結構上與函式呼叫點同構，其中呼叫者綁定引數，被呼叫者回傳在其他地方計算的值，而後續程式碼則消費該值。這種條件結構在普通程式碼中普遍存在於網路規模。我們透過函式感知的中間填充（FIM）中間訓練來利用這一點：這是一種自監督目標，它遮罩透過程式依賴圖分析以及複雜度-可推論性雙重準則所選出的函式。我們在來自968個GitHub儲存庫、含有26億個詞元的去污染語料庫上對Qwen2.5-Coder-Instruct（7B/14B）和Qwen3-8B進行中間訓練，然後應用現有的代理後訓練流程。中間訓練使SWE-Bench-Verified在7B/14B上分別提升+2.8/+3.0，在Qwen3-8B上提升+3.2；SWE-Bench-Lite在同模型上分別獲得+3.7/+4.0/+5.4的提升。此改善在兩個後訓練流程（R2E-Gym、SWE-Smith）以及非Qwen2.5基礎模型（搭配SWE-Lego的Qwen3-8B）上均保持一致。除了領域內的提升之外，中間訓練還緩解了代理後訓練原本會對非代理編碼（例如LiveCodeBench）以及非編碼工具使用基準（tau-bench、BFCL）造成的能力侵蝕：儘管中間訓練語料庫僅包含Python程式碼，但函式呼叫的歸納偏置在後訓練中得以保留，並產生一致的提升。

English

Coding agents must integrate external tool returns into ongoing reasoning - a capability that standard left-to-right pretraining on code exposes only in its forward direction. We observe that the action-observation-continuation loop of a coding agent is structurally isomorphic to a function call site, where a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. This conditioning structure exists at internet scale in ordinary code. We exploit it through function-aware fill-in-the-middle (FIM) mid-training: a self-supervised objective that masks functions selected via program dependency graph analysis and a complexity-inferability double criterion. We mid-train Qwen2.5-Coder-Instruct (7B/14B) and Qwen3-8B on a 2.6B-token decontaminated corpus drawn from 968 GitHub repositories, then apply existing agentic post-training pipelines. Mid-training improves SWE-Bench-Verified by +2.8/+3.0 at 7B/14B and by +3.2 on Qwen3-8B; SWE-Bench-Lite gains are +3.7/+4.0/+5.4 on the same models. The improvement holds across two post-training pipelines (R2E-Gym, SWE-Smith) and on a non-Qwen2.5 base (Qwen3-8B with SWE-Lego). Beyond in-domain gains, mid-training also mitigates the capability erosion that agentic post-training otherwise inflicts on non-agent coding (e.g., LiveCodeBench) and non-coding tool-use benchmarks (tau-bench, BFCL): although the mid-training corpus contains Python code only, the function-call inductive bias survives post-training and yields consistent gains.