基於智能體經驗的樣本高效學習

摘要

現實世界中的智能體學習往往受限於昂貴的環境互動，例如執行耗時的實驗或獲取人類回饋。情境學習提供一種高度樣本效率的方式，讓智能體能從自身的互動歷史中學習，但一旦該經驗從情境中被移除，其效益便會消失。另一方面，情境蒸餾提供了一種將情境資訊內化至模型權重的機制。然而，將其應用於智能體的互動歷史且不犧牲環境樣本效率，目前仍未被充分探討。我們將此問題稱為「經驗蒸餾」，並開發出一種無需在已收集經驗之外進行額外環境互動的實作方法。在749個精選軟體工程任務與六個文字冒險遊戲上的實驗顯示，該方法能保留至少64.8%來自情境學習的增益，而直接對收集到的經驗進行監督式微調僅能恢復3.8%的增益。相較於傳統的強化學習基準方法，透過嘗試-錯誤經驗進行情境學習再搭配經驗蒸餾，能以至少9.6倍的環境樣本減少達到相同的效能。

English

Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8\% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8\%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least \(9.6\times\) fewer environment samples.