ContextPilot：透過細粒度強化學習教導智能體進行主動式上下文管理

摘要

長視域代理任務要求大型語言模型（LLMs）在多輪互動中反覆檢索、整合並維護分散的資訊，但保留所有互動歷史會導致工作上下文持續增長。近期的主動式上下文管理方法允許模型透過專用工具編輯自身的工作上下文，但仍面臨三個關鍵限制：（1）工具集有限，僅限於搜尋、刪除和摘要，不支援全局規劃、長期記憶和自適應壓縮；（2）探索效率低下，將上下文管理動作視為同質處理，儘管它們對最終結果的影響具有異質性；（3）粗粒度的信用分配，在強化學習期間將最終的軌跡層級獎勵分配給所有中間的上下文編輯動作。為填補這些差距，我們提出了ContextPilot，一個用於長視域代理推理的主動式上下文管理框架。我們的方法系統性地擴充工具集，納入規劃、長期記憶和軟性上下文卸載工具。我們進一步提出了一種專為上下文管理設計的強化學習方法，該方法利用上下文與熵的變化來識別關鍵的編輯決策以進行分支採樣，並從所有通過對應上下文編輯動作的分支軌跡中估計動作層級優勢。在長上下文問答和深度搜索任務上的實驗表明，ContextPilot以更緊湊的工作上下文實現了更強的效能，在各種基礎模型和基準上 consistently 優於現有基線方法。程式碼可在 https://github.com/Tencent/ContextPilot 取得。

English

Long-horizon agentic tasks require large language models (LLMs) to iteratively retrieve, integrate, and maintain dispersed information across multi-turn interactions, but preserving all interaction histories leads to a continuously growing working context. Recent proactive context management methods allow models to edit their own working context with specialized tools, yet they still face three key limitations: (1) a limited toolset restricted to search, deletion, and summarization, with no support for global planning, long-term memory, and adaptive compression; (2) inefficient exploration that treats context management actions uniformly despite their heterogeneous impacts on final outcomes; and (3) coarse-grained credit assignment that assigns the final trajectory-level reward to all intermediate context editing actions during RL. To bridge these gaps, we introduce ContextPilot, a proactive context management framework for long-horizon agentic reasoning. Our approach systematically augments the toolset with planning, long-term memory, and soft context offloading tools. We further propose an RL method tailored for context management, which uses context and entropy variation to identify critical editing decisions for branch sampling and estimates action-level advantages from all branched trajectories that pass through the corresponding context editing action. Experiments on long-context QA and deep search tasks show that ContextPilot achieves stronger performance with a more compact working context, consistently outperforming existing baselines across various base models and benchmarks. Code is available at https://github.com/Tencent/ContextPilot.