QuoteBench：匹配分數如何隱藏指令路徑失敗

摘要

大型語言模型程式碼代理透過可能對模型輸出進行序列化、包裝和重新解析的介面發出 Bash 指令。僅靠配對執行分數無法區分指令生成錯誤與生成後引入的失敗。QuoteBench 透過在來自 14 個源自事件的系列中的 56 個單次任務上進行精確的最終狀態驗證來衡量此邊界，並圍繞一個刻意未跳脫的附加解析器，將生成契約與執行傳輸層交叉結合。在插值點進行跳脫可重現每個重播回覆的原始路徑結果，因此在揭露的邊界下任何恢復都必須來自模型改變其生成內容。在八種相同視窗配置中，透過附加解析器重播相同回覆會使成功率降低 55.4 至 73.2 個百分點；揭露邊界對六種配置恢復了 30.4 至 60.7 個百分點，對另外兩種配置則恢復為零或略微負值。原始生成在前緣已接近飽和；邊界適應才是仍然區分模型表現的因素。GPT-5.6-sol 的 -3.6 點配對差距隱藏了 -64.3 點的損害和 +60.7 點的補償。部署配置會重新排序模型：26 個可比較配對中有一個逆轉是明確的，另有四個處於單一任務的邊際上。對發出指令的代理進行評估時，應報告模型配置、生成契約、執行路徑、操作點和最終狀態驗證器，而不應將配對分數視為模型的內在屬性。

English

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.