S3Gym：大语言模型能否将自我测试与自我评判转化为自我改进？

摘要

大语言模型（LLMs）正越来越多地与外部环境交互，并在此过程中积累大量行为经验。然而，现有的智能体基准大多将它们作为固定策略进行评估，因此，智能体能否主动检验自身行为、评判由此产生的经验，并利用这些经验改进未来的决策，仍是一个悬而未决的问题。我们提出 S³Gym——一个通过三种相互耦合的能力（自我测试、自我评判与自我改进）来评估 LLM 自我改进能力的交互式基准。S³Gym 将宽松式探索与严格的留出式评估相分离，并在七个具备可执行环境验证器的文本游戏中实例化这一协议。我们评估了三种融入交互经验的路径：基于历史记录的上下文学习（History ICL）、以分数为条件的摘要记忆（Score-conditioned Summary Memory）以及参数训练（Parameter Training）。实验结果表明，自我改进既非自动实现，也非整齐划一。上下文层面的经验提升了若干“模型-游戏”组合的性能，但最有效的路径在很大程度上取决于任务结构：当经验可以被压缩为可复用的策略性规则时，摘要更具优势；而当成功依赖于精确的、因状态而异的信息时，摘要往往不及原始历史记录。参数训练在某些任务上带来了显著提升，但在其他任务上则表现出不稳定的改进以及严重的负迁移。这些发现表明，仅仅识别出成功的行动并不足够；智能体还必须将反馈转化为可执行且可迁移的策略。S³Gym 为诊断这一过程、识别阻碍智能体将交互经验转化为可靠自我改进的瓶颈，提供了一个统一框架。

English

Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce S\textsuperscript{3Gym}, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: Self-Testing, Self-Judging, and Self-Improvement. S^3Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S^3Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.