经验链用于大语言模型的持续改进

摘要

人类不断从经验中学习，而传统大型语言模型（LLM）评估忽略了模型通过推理时交互进行改进的能力。在本文中，我们研究LLM如何在测试时从迭代经验中学习，我们将这一设定称为经验链（Chain-of-Experience, CoE），其中模型通过自我反馈或环境反馈的迭代交互积累经验轨迹，形成超越零样本推理的持续改进循环。我们通过多种反馈机制实例化CoE，包括模型自我反馈和环境信号（如正确性或公开编程测试通过率），并使用8个LLM（包括GPT-5、Gemini-2.5 Pro、Claude-4.5 Sonnet）在数学、编程和知识领域进行评估。我们的研究表明，利用迭代经验始终优于无反馈基线，仅凭自我反馈即可实现显著提升，同时在各任务和模型上总体改进5.6%，API成本降低19%。我们进一步表明，结合互补反馈通道（如模型信号和正确性信号）能带来额外收益，并且CoE比现有测试时策略实现更高的每token准确率。我们观察到LLM基础能力与改进能力之间存在正相关，并表明模型在弱反馈或虚假反馈下仍保持鲁棒性，不同反馈有助于不同的改进方面，且大部分收益出现在迭代早期。

English

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.