連點成線：透過強化學習訓練具備跨域泛化能力之長期生命週期代理的大型語言模型

摘要

本研究提出了一個通用框架，用於訓練大型語言模型（LLMs）具備「串連線索」（Connect the Dots, CoD）的能力——這是長期生命週期代理所需的一種後設能力：當基於LLM的AI代理部署在環境中時，它會解決一系列長期任務，同時持續探索環境、從自身經驗中學習，並反覆自我更新關於環境的上下文，從而在更新後的上下文條件下，在未來任務中逐步獲得更好的表現。CoD框架的主要組成部分包括：（1）演算法設計與基礎架構，用於端到端強化學習（RL），其中交錯進行任務解決與上下文更新的長序列展開；（2）任務與環境，用於在訓練過程中激勵並引出LLM中目標後設能力，以及在評估過程中準確衡量進展。我們提供了CoD框架的概念驗證實作，包括具有細粒度信用分配的GRPO風格RL演算法，以及針對目標後設能力（而非特定領域的LLM能力或標準的逐任務RL）量身訂製的任務與環境。實證結果驗證了在CoD設定下進行端到端RL訓練的有效性，並展示了所引出的後設能力在分佈外泛化方面的潛力——包括在訓練領域內、跨不同領域，以及從CoD到Ralph-loop設定。我們對CoD的研究連結了多條先前工作，並為推進LLM與AI代理開創了新的機會。為促進進一步研究與應用，我們在https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod 開源了實作。

English

This work presents a general framework for training large language models (LLMs) to "Connect the Dots" (CoD), a meta-capability required by long-lifecycle agents: as an LLM-based AI agent gets deployed in an environment, it solves a long sequence of tasks while continuously exploring the environment, learning from its own experiences, and iteratively self-updating its context about the environment, thereby achieving progressively better performance on future tasks conditioned on the updated context. Major components of the CoD framework include: (1) algorithm design and infrastructure for end-to-end reinforcement learning (RL) with long rollout sequences interleaving solve-task and update-context episodes; (2) tasks and environments for incentivizing and eliciting the targeted meta-capability in LLMs during training, as well as for faithfully measuring progress during evaluation. We present proof-of-concept implementations of the CoD framework, including a GRPO-style RL algorithm with fine-grained credit assignment, as well as tasks and environments tailored to the targeted meta-capability (rather than domain-specific LLM capabilities or standard task-by-task RL). Empirical results validate the efficacy of end-to-end RL training in the CoD setting, and demonstrate the potential for out-of-distribution generalization -- within the training domains, across different domains, and from CoD to Ralph-loop settings -- of the elicited meta-capability. Our investigation of CoD connects several lines of prior works, and opens up new opportunities for advancing LLMs and AI agents. To facilitate further research and applications, we release our implementations at https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod.