SpecFirst: 将行为规范提取作为从零开始进行基于智能体的程序合成的首要步骤

摘要

基于LLM的智能体在利用现有代码库提供上下文的软件工程任务中表现出色，但从头开始构建程序本质上更为困难。近期如ProgramBench等基准测试量化了这一差距：仅给定自然语言文档和仅可执行的二进制文件作为行为预言机，即便是最前沿的模型也只能解决不到1%的实例。现有框架将文档阅读、行为探索和代码合成合并为单一流程，导致智能体探索不充分、上下文偏移时行为意图丢失，以及早期误解被带入最终实现。受经典需求工程启发，我们认为行为规格获取应作为实现前的首要阶段。我们提出SpecFirst，一种两阶段框架，强制在代码合成之前进行规格获取。专用规格智能体首先探测二进制文件，将观测结果与文档结合形成结构化规格说明。随后，代码合成智能体利用该规格说明驱动实现。这种分解在编码开始前解决了文档歧义问题，并在整个合成过程中提供稳定的行为参考。我们在所有200个ProgramBench实例上对SpecFirst进行评估，涉及两个模型家族及一个数量级能力差异的四种模型。SpecFirst始终优于单循环基线方法，测试通过率提升6.9%至21.3%，二进制探索覆盖率提升9.4%至18.5%，均具有统计显著性。对代码合成的行为分析进一步表明，先行的规格说明能实现更早且更持续的结构构建。我们的结果表明，明确的需求工程阶段是一种有效的从头开始程序构建范式。

English

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.