分层自我改进：面向任务特定可进化智能体支架的框架

摘要

现代大语言模型智能体通常通过手动修改提示、工具或工作流来改进，而环绕模型的可执行骨架——即执行框架（harness）——在部署后往往被视为固定产物。本研究探讨一种替代方案：执行框架是任务特定且可持续演化的——每个任务族维护各自的执行框架，该框架通过固定的任务注入接缝在迭代间热切换，并利用环境反馈进行重写。我们提出分层自我改进（Hierarchical Self-Improvement, HSI）框架，其中单个冻结的大语言模型M在三个分层范围内运作：执行任务的任务执行框架H、重写H的演化器（evolver），以及在冻结外部锚点下重写演化器策略代码的元演化器（meta-evolver）。一种推理开关（thinking-on/off）设计通过在执行任务时禁用推理、在自我修改时启用推理，从而隔离执行框架演化的贡献。HSI受两个因素制约：反馈保真度界限——因为演化需要信息丰富的奖励信号来引导选择；以及骨干模型能力界限——因为执行框架的重设计无法克服冻结模型的局限。在BALROG基准上，以DeepSeek-V4-Flash-Preview作为冻结骨干模型，HSI在中等难度任务上相较初始执行框架取得了一致的提升（BabyAI上+39.3，Crafter上+33.0，TextWorld上+25.0，MiniHack上+15.0，均为原始%进度），同时在BabaIsAI子套件上获得了强大的留出泛化性能（在20%未见数据划分下，BreakStop最佳测试分数为0.98，GoTo为1.00）。在超出骨干模型能力的任务（NLE）上，执行框架演化未带来改进。这些结果表明，任务特定的执行框架演化是改进冻结大语言模型智能体的一个可行方向，但存在明确的实证界限。代码见 https://github.com/TailinZhou/hsi。

English

Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the harness---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is task-specific and continuously evolvable: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce Hierarchical Self-Improvement (HSI), a framework in which a single frozen LLM M operates across three hierarchical scopes: a task harness H that executes tasks, an evolver that rewrites H, and a meta-evolver that rewrites the evolver's strategy code under a frozen outer anchor. A thinking-on/off design isolates the contribution of harness evolution by disabling reasoning during task execution while enabling it during self-modification. HSI is bounded by two factors: a feedback-fidelity bound, since evolution requires informative reward signals to guide selection, and a backbone capability bound, since harness redesign cannot overcome limitations of the frozen model. On BALROG with DeepSeek-V4-Flash-Preview as the frozen backbone, HSI achieves consistent gains over the initial harness on moderate-difficulty tasks (+39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, and +15.0 on MiniHack, all in raw \% Progress), while obtaining strong held-out generalization on BabaIsAI sub-suites (0.98 best-test on BreakStop and 1.00 on GoTo from a 20% unseen split). On tasks beyond the backbone's capability (NLE), harness evolution provides no improvement. These results demonstrate task-specific harness evolution as a viable axis for improving frozen LLM agents under clear empirical limits. Code is available at https://github.com/TailinZhou/hsi.