基于智能体经验的样本高效学习

摘要

现实世界中的智能体学习往往受限于高昂的环境交互成本，例如运行耗时实验或获取人工反馈。上下文学习为智能体提供了一种高度样本高效的方式，使其能从自身的交互历史中学习，但一旦该经验从上下文中移除，其收益便会消失。另一方面，上下文蒸馏提供了一种将上下文信息内化到模型权重中的机制。然而，如何将其应用于智能体的交互历史而同时不牺牲环境样本效率，这一问题仍未被充分探索。我们将此问题称为经验蒸馏，并开发了一种无需在已收集经验之外进行更多环境交互的实现方法。在749个精心策划的软件工程任务和六个文字冒险游戏上的实验表明，该方法在两个领域中至少保留了上下文学习收益的64.8%，而直接对收集到的经验进行监督微调仅能恢复3.8%的收益。与经典强化学习基线相比，基于试错经验的上下文学习后接经验蒸馏，在环境样本数量至少减少9.6倍的情况下达到了与其相当的性能。

English

Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8\% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8\%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least \(9.6\times\) fewer environment samples.