Agentic ESOpt：以最低GPU需求微調長程大型語言模型智能體

摘要

強化學習（Reinforcement Learning, RL）在單回合大型語言模型微調上已展現潛力。然而，長時域代理式推理引入了日益分支化的互動與稀疏獎勵，暴露了 RL 的多項限制：其基於反向傳播的重量級訓練技術棧使得微調較大型語言模型不切實際，且更長的軌跡使 RL 中的信用分配變得更加困難。本文主張演化策略（Evolution Strategies, ES）可成為微調長時域語言模型代理的更好選擇。與代理式 RL 相比，ES 具有三項關鍵優勢：1) 模型可擴展性：ES 僅需最少、推論等級的 GPU 記憶體即可進行全參數最佳化，使得微調大型語言模型成為可能。2) 靈活性：其輕量、黑箱的回饋介面使 ES 微調易於與提示空間演化（例如技能最佳化與測試時計算）組合；3) 長時域可擴展性：ES 在不跨時域分解獎勵的情況下執行軌跡等級的參數歸因，隨著時域長度增長，其可擴展性優於代理式 RL。基於此洞察，我們提出 Agentic ESOpt，一個專為彈性參數–上下文共同演化而設計的全參數代理式微調框架。在每一步，Agentic ESOpt 在當前語言模型參數周圍採樣擾動，以獎勵評估所產生的代理，並應用線上獎勵加權更新。為改善探索–適應的取捨，Agentic ESOpt 進一步引入擾動尺度 σ 的餘弦衰減排程。在 WebArena-Lite 上，對 Qwen-3.5-27B 進行全參數最佳化將 No Skill 基準提升了 6.69%。在測試時自動啟發式設計中，Agentic ESOpt 執行線上提示–參數共同演化，在 36 個設定中的 28 個改善了其對應基準。

English

Reinforcement Learning (RL) has been promising in single-turn LLM fine-tuning. However, long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing several limitations of RL: its heavyweight backpropagation-based training stack makes it impractical to fine-tune larger LLMs, and longer-horizon trajectories make credit assignment in RL substantially harder. This paper argues that evolution strategies (ES) can be a better choice for fine-tuning long-horizon LLM agents. Compared with agentic RL, ES offers three key advantages: 1) Model Scalability: ES enables full-parameter optimization with only minimal, inference-level GPU memory, making it possible to fine-tune large LLMs. 2) Flexibility: its lightweight, black-box feedback interface makes ES fine-tuning easy to compose with prompt-space evolution (e.g., skill optimization & test-time compute); and 3) Long-Horizon Scalability: ES performs trajectory-level parameter attribution without decomposing rewards across horizons, yielding better scalability than Agentic RL as the horizon length grows. Based on this insight, we propose Agentic ESOpt, a full-parameter agentic fine-tuning framework tailored to flexible parameter--context co-evolution. At each step, Agentic ESOpt samples perturbations around the current LLM parameters, evaluates the resulting agents with rewards, and applies an online reward-weighted update. To improve the exploration--adaptation trade-off, Agentic ESOpt further introduces a cosine decay schedule of the perturbation scale σ. On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improves the No Skill baseline by 6.69%. In test-time automatic heuristic design, Agentic ESOpt performs online prompt--parameter co-evolution, improving its matched baseline in 28 of 36 settings.