无知还是无能？为大语言模型智能体构建知识门控的可验证任务

摘要

专业智能体任务往往依赖公共语料库中所没有的约定，但基准测试很少控制智能体能否访问这些约定。我们提出了一种知识门控的任务构建协议，将任务指令与一个紧凑工件相分离，该工件包含私有约定、参考表和实用算子。构建时的溯源、在提供与不提供工件两种条件下逐字节完全相同的任务指令、泄漏审计以及可执行见证，使得任务对工件的依赖变得明确且可检验。在十五个校准任务中，某一种前沿智能体配置在提供工件时取得68.0%的通过率，而在不提供工件时为0%；在其中一个任务上，即便提供看似合理但不正确的工件，五次试验的通过率也仍为0%。确定性求解器和规则语料库为结构化任务提供精确的真值；无法由单一可执行预言机检查的输出，则依靠具名的标准级评分量规来评估。经配置相对的校准筛选，我们保留了七个满足五次试验经验知识门控筛选要求的任务。这些实验验证了构建协议的行为，但并未证明所保留的任务能改善后训练。我们已在 https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction 公开发布该任务套件的部分内容及配套工具。

English

Professional agent tasks often depend on conventions that are absent from public corpora, yet benchmarks rarely control whether an agent has access to those conventions. We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact containing private conventions, reference tables, and utility operators. Construction-time provenance, byte-identical task instructions across the provided- and withheld-artefact conditions, leak audits, and executable witnesses make dependence on the artefact explicit and testable. Across fifteen calibration tasks, one frontier agent configuration achieves a 68.0% pass rate with the artefact and 0% without it; on one task, a plausible but incorrect artefact also yields 0% across five trials. Deterministic solvers and rule corpora provide exact ground truth for structured tasks, while named criterion-level rubrics support outputs that cannot be checked by a single executable oracle. A configuration-relative calibration screen retains seven tasks satisfying our five-trial empirical knowledge-gating screen. These experiments validate the behavior of the construction protocol; they do not establish that the retained tasks improve post-training. We publicly release part of the task suite and supporting tooling at https://github.com/DatagridsAI/Knowledge-Gated-Task-Construction.