ContextPilot：利用细粒度强化学习教智能体进行主动上下文管理

摘要

长程智能体任务要求大语言模型在多轮交互中迭代式地检索、整合和维护分散的信息，但保留全部交互历史会导致工作上下文持续增长。现有的主动上下文管理方法允许模型通过专门工具编辑自身的工作上下文，但仍存在三个关键局限：(1) 受限的工具集，仅支持搜索、删除和摘要，缺乏全局规划、长期记忆和自适应压缩能力；(2) 低效的探索方式，对所有上下文管理动作一视同仁，尽管它们对最终结果具有差异性影响；(3) 粗粒度的信用分配，在强化学习过程中将最终的轨迹级奖励分配给所有中间上下文编辑动作。为弥补上述不足，我们提出ContextPilot——一个面向长程智能体推理的主动上下文管理框架。该方法系统性地扩展了工具集，新增规划、长期记忆和软上下文卸载工具。我们进一步提出一种针对上下文管理定制的强化学习方法，利用上下文和熵的变化识别关键编辑决策以进行分支采样，并从所有经过相应上下文编辑动作的分支轨迹中估计动作级优势。在长上下文问答和深度搜索任务上的实验表明，ContextPilot以更紧凑的工作上下文取得了更强的性能，在多种基座模型和基准上持续超越现有基线。代码已开源：https://github.com/Tencent/ContextPilot。

English

Long-horizon agentic tasks require large language models (LLMs) to iteratively retrieve, integrate, and maintain dispersed information across multi-turn interactions, but preserving all interaction histories leads to a continuously growing working context. Recent proactive context management methods allow models to edit their own working context with specialized tools, yet they still face three key limitations: (1) a limited toolset restricted to search, deletion, and summarization, with no support for global planning, long-term memory, and adaptive compression; (2) inefficient exploration that treats context management actions uniformly despite their heterogeneous impacts on final outcomes; and (3) coarse-grained credit assignment that assigns the final trajectory-level reward to all intermediate context editing actions during RL. To bridge these gaps, we introduce ContextPilot, a proactive context management framework for long-horizon agentic reasoning. Our approach systematically augments the toolset with planning, long-term memory, and soft context offloading tools. We further propose an RL method tailored for context management, which uses context and entropy variation to identify critical editing decisions for branch sampling and estimates action-level advantages from all branched trajectories that pass through the corresponding context editing action. Experiments on long-context QA and deep search tasks show that ContextPilot achieves stronger performance with a more compact working context, consistently outperforming existing baselines across various base models and benchmarks. Code is available at https://github.com/Tencent/ContextPilot.