Chain-of-Experience：用於大型語言模型持續改進的經驗鏈方法

摘要

人類持續從經驗中學習，而傳統的大型語言模型（LLM）評估方式卻忽略了模型透過推論時互動來提升能力。在本論文中，我們研究LLM如何在測試階段從迭代經驗中學習，我們將此設定稱為「經驗鏈」（Chain-of-Experience, CoE），在此設定下，模型透過與自身或環境回饋的迭代互動累積經驗軌跡，形成超越零樣本推論的持續改進循環。我們以多樣化的回饋機制實例化CoE，包括模型自我回饋以及正確性判斷或公開程式碼測試通過率等環境訊號，並使用8個LLM（包括GPT-5、Gemini-2.5 Pro、Claude-4.5 Sonnet）在數學、程式設計和知識領域進行評估。我們的研究顯示，利用迭代經驗 consistently 優於無回饋的基線方法，僅憑自我回饋即能獲得顯著提升，且在各種任務和模型上整體提升5.6%，API成本降低19%。我們進一步證明，結合互補的回饋管道（例如模型訊號與正確性訊號）能帶來額外增益，且CoE在每個token的準確率上優於現有的測試時策略。我們觀察到LLM的基礎能力與改進能力之間存在正相關，並顯示模型在弱回饋或虛假回饋下仍能保持穩健性，不同的回饋有助於不同的改進面向，且大多數增益在迭代初期即已出現。

English

Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.