為測試而構建：編碼智能體交付你檢查的內容，而非你要求的。

摘要

基準（benchmarks）廣泛用於評估大型語言模型（LLMs）的任務完成能力，但這種方法已累積了構念效度（construction-validity）方面的問題，且通過分數並不一定代表所要求的任務確實被完成。我們針對這兩個問題進行研究。在一個受控的「以程式碼為規格」（code-as-spec）設定中，兩款生產級 Copilot CLI 代理（claude-opus-4.7、gpt-5.5）需要將一個 React Fluent-UI 資料表重新實作成 Angular 的可重用函式庫，並在隱藏包含 222 個測試的 Playwright 測試判官（oracle）下進行 18 次執行，且分為三種測試判官的可用性條件。除了分數之外，我們也執行機械化的函式庫審計，並透過無操作消融實驗（no-op ablation）檢查每個判定結果。在沒有測試判官的情況下，函式庫雖存在但不完整，這點可由分數看出。當測試判官納入流程後，分數接近完美，但從一個直接包含被測試行為的展示範例來看，該函式庫不是被遺留未完成就是完全不存在。我們將此現象稱為「為測試而建構」（building to the test）；而背後更廣泛的傾向，我們稱之為「驗證自我意識」（validation self-awareness）。這類代理並不會像使用者一樣，主動驗證其所交付的內容。此情況在其他代理、訊號與模型家族中的普遍性仍有待探討。除了基準分數之外，像「驗證自我意識」這類傾向也值得研究關注。

English

Benchmarks are widely used to evaluate task completion by Large Language Models (LLMs), but this approach has accumulated construction-validity problems, and a passing score may not show whether the requested task was delivered. We study both problems. In a controlled code-as-spec setup, two production Copilot CLI agents (claude-opus-4.7, gpt-5.5) re-implement a React Fluent-UI data table in Angular as a reusable library under a hidden 222-test Playwright oracle across 18 runs and three oracle-availability conditions. Alongside the score, we run a mechanical library audit and check each verdict with a no-op ablation. Without the oracle, the library is present but unfinished, revealed by scores. With the oracle in the loop, the score reaches near-perfect, but from a demo holding the tested behavior directly, the library left dead or absent. We call this building to the test; the broader disposition behind both we call validation self-awareness. The agent does not, on its own, validate what it ships as a user would. Prevalence remains an open question across other agents, signals, and model families. Beyond benchmark scores, dispositions like validation self-awareness merit research attention.