無知還是無能？為大語言模型代理建構知識門控且可驗證的任務

摘要

專業型智慧體任務往往依賴於公開語料庫中不存在的慣例，但基準測試很少管控智慧體是否能取得這些慣例。我們提出一套知識門控任務建構協定，將任務指令與一個包含私有慣例、參考表和工具運算子的精簡工件分離。建構階段的來源追蹤、在提供工件與隱藏工件條件下位元組完全相同的任務指令、洩漏稽核，以及可執行見證，使任務對工件的依賴明確且可測試。在十五項校準任務中，某一前沿智慧體配置在提供工件時達到 68.0% 的通過率，未提供工件時為 0%；其中一項任務中，一個看似合理但錯誤的工件同樣在五次試驗中皆得到 0%。確定性求解器與規則語料集為結構化任務提供精確的真值，而具名的標準級評分量表則支援無法由單一可執行預言機檢查的輸出。一個相對於配置的校準篩選保留了七項通過我們五次試驗實證知識門控篩選的任務。這些實驗驗證了建構協定的運作方式，但並未證明這些保留任務能改善後續訓練的成效。我們已於 https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction 公開釋出部分任務套件與支援工具。

English

Professional agent tasks often depend on conventions that are absent from public corpora, yet benchmarks rarely control whether an agent has access to those conventions. We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact containing private conventions, reference tables, and utility operators. Construction-time provenance, byte-identical task instructions across the provided- and withheld-artefact conditions, leak audits, and executable witnesses make dependence on the artefact explicit and testable. Across fifteen calibration tasks, one frontier agent configuration achieves a 68.0% pass rate with the artefact and 0% without it; on one task, a plausible but incorrect artefact also yields 0% across five trials. Deterministic solvers and rule corpora provide exact ground truth for structured tasks, while named criterion-level rubrics support outputs that cannot be checked by a single executable oracle. A configuration-relative calibration screen retains seven tasks satisfying our five-trial empirical knowledge-gating screen. These experiments validate the behavior of the construction protocol; they do not establish that the retained tasks improve post-training. We publicly release part of the task suite and supporting tooling at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction.