層次化自我改進：一種針對特定任務可演化智能體控制框架

摘要

現代LLM代理通常透過手動修改提示、工具或工作流程來改進，而圍繞模型的可執行支架——即執行框架——在部署後通常被視為固定產物。本研究探討另一種做法：執行框架是任務特定且可持續演化的。每個任務系列維護自己的執行框架，透過固定的任務注入接口在迭代間熱插拔，並利用環境反饋重新編寫。我們提出分層自我改進（HSI），一個單一凍結LLM M在三種層級範圍內運作的框架：執行任務的任務執行框架H、重寫H的演化器，以及在凍結的外部錨點下重寫演化器策略程式碼的元演化器。思考開啟/關閉設計透過在任務執行期間停用推理、在自我修改期間啟用推理，隔離了執行框架演化的貢獻。HSI受到兩個因素限制：反饋保真度界限——因為演化需要資訊豐富的獎勵訊號來引導選擇；以及骨幹能力界限——因為執行框架的重新設計無法克服凍結模型的限制。在BALROG上，以DeepSeek-V4-Flash-Preview作為凍結骨幹，HSI在中度困難任務上相較初始執行框架取得一致的提升（BabyAI +39.3，Crafter +33.0，TextWorld +25.0，MiniHack +15.0，均為原始進度百分比），同時在BabaIsAI子套件上獲得強勁的留出泛化（來自20%未見分割的BreakStop最佳測試成績0.98，GoTo為1.00）。在超出骨幹能力的任務（NLE）上，執行框架演化沒有提供改進。這些結果表明，任務特定的執行框架演化是在明確的經驗限制下改進凍結LLM代理的一個可行方向。程式碼可於 https://github.com/TailinZhou/hsi 取得。

English

Modern LLM agents are often improved by modifying prompts, tools, or workflows manually, while the executable scaffold surrounding the model---the harness---is typically treated as a fixed artifact after deployment. This work studies an alternative where the harness is task-specific and continuously evolvable: each task family maintains its own harness, which is hot-swapped across iterations through a fixed task-injection seam and rewritten using environment feedback. We introduce Hierarchical Self-Improvement (HSI), a framework in which a single frozen LLM M operates across three hierarchical scopes: a task harness H that executes tasks, an evolver that rewrites H, and a meta-evolver that rewrites the evolver's strategy code under a frozen outer anchor. A thinking-on/off design isolates the contribution of harness evolution by disabling reasoning during task execution while enabling it during self-modification. HSI is bounded by two factors: a feedback-fidelity bound, since evolution requires informative reward signals to guide selection, and a backbone capability bound, since harness redesign cannot overcome limitations of the frozen model. On BALROG with DeepSeek-V4-Flash-Preview as the frozen backbone, HSI achieves consistent gains over the initial harness on moderate-difficulty tasks (+39.3 on BabyAI, +33.0 on Crafter, +25.0 on TextWorld, and +15.0 on MiniHack, all in raw \% Progress), while obtaining strong held-out generalization on BabaIsAI sub-suites (0.98 best-test on BreakStop and 1.00 on GoTo from a 20% unseen split). On tasks beyond the backbone's capability (NLE), harness evolution provides no improvement. These results demonstrate task-specific harness evolution as a viable axis for improving frozen LLM agents under clear empirical limits. Code is available at https://github.com/TailinZhou/hsi.