QuoteBench：匹配分数如何掩盖命令路径失败

摘要

LLM 编码代理通过可能对模型输出进行序列化、包装和重新解析的接口发出 Bash 命令。仅靠匹配执行分数无法区分命令生成错误与生成后引入的失败。QuoteBench 通过在源自 14 个事件的 56 个单次任务上进行精确最终状态验证来测量这一边界，将生成契约与执行传输在一个故意未转义的附加解析器处交叉。在插值点进行转义可重现每个重放回复的原始路径结果，因此在披露边界下的任何恢复都必须来自模型改变其生成。在八种同一窗口配置中，通过附加解析器重放相同回复会使成功率降低 55.4 至 73.2 个百分点；披露可为六种配置恢复 30.4 至 60.7 个百分点，而另外两种配置的恢复为零或略为负。原始生成在前沿已接近饱和；边界适应才是仍然区分模型的要素。GPT-5.6-sol 的匹配差距为 -3.6 个百分点，掩盖了 -64.3 个百分点的损害与 +60.7 个百分点的补偿。部署配置会重新排列模型顺序：26 个可比较对中有一个明确反转，另有四个处于单任务边缘。对命令发出代理的评估应报告模型配置、生成契约、执行路径、操作点和最终状态验证器，而不应将匹配分数视为模型的内在属性。

English

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.