S3Gym：大型語言模型能否將自我測試與自我評判轉化為自我改進？

摘要

大型語言模型（LLMs）日益與外部環境互動並累積大量行為經驗，然而現有的智能體基準測試大多將其視為固定策略來評估。因此，智能體能否主動測試自身行為、判斷由此產生的經驗，並利用這些經驗來改善未來的決策，仍是個未解的問題。我們提出S³Gym，一個交互式基準測試，通過三種相互耦合的能力來評估大型語言模型的自我改進：自我測試（Self-Testing）、自我判斷（Self-Judging）與自我改進（Self-Improvement）。S³Gym將允許自由探索的環境與嚴格、預先保留的評估環境分開，並在七個帶有可執行環境驗證器的文字遊戲中實例化此協議。我們評估了三種整合互動經驗的途徑：直接的歷史上下文學習（History ICL）、基於分數條件的摘要記憶（Summary Memory），以及參數訓練（Parameter Training）。我們的實驗揭示，自我改進既非自動也非一致。上下文層級的經驗能提升若干模型—遊戲配對的表現，但最有效的途徑在很大程度上取決於任務結構：當經驗能被壓縮成可重用的戰略規則時，摘要方法有益；然而當成功依賴於精確的、依狀態而定的資訊時，摘要往往不如原始歷史紀錄。參數訓練在某些任務上帶來顯著增益，但在其他任務上則表現出不穩定的改進以及嚴重的負遷移。這些發現表明，僅僅識別成功的行動是不夠的；智能體還必須將回饋轉化為可執行且可遷移的策略。S³Gym提供了一個統一的框架，用以診斷這一過程，並找出阻礙智能體將互動經驗轉化為可靠自我改進的瓶頸。

English

Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience, yet existing agent benchmarks largely evaluate them as fixed policies. It therefore remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions. We introduce S\textsuperscript{3Gym}, an interactive benchmark for evaluating LLM self-improvement through three coupled capabilities: Self-Testing, Self-Judging, and Self-Improvement. S^3Gym separates permissive exploration from strict held-out evaluation and instantiates this protocol in seven text-based games with executable environment verifiers. We evaluate three pathways for incorporating interaction experience: direct History ICL, score-conditioned Summary Memory, and parameter Training. Our experiments reveal that self-improvement is neither automatic nor uniform. Context-level experience improves performance for several model--game pairs, but the most effective pathway depends strongly on the task structure: summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when success depends on precise, state-contingent information. Parameter training produces substantial gains on some tasks, but also exhibits unstable improvement and severe negative transfer on others. These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies. S^3Gym provides a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable self-improvement.