获取、修复、保持：面向小模型对话游戏智能体的诊断引导式后训练方法

摘要

交互式对话游戏考验的是静态基准在很大程度上隐式保留的一种能力：模型必须在多轮对话中保持状态、解读反馈，并在不断变化的约束条件下选择有效动作。我们在LM Playschool挑战赛中使用一个20亿参数开放权重模型研究这一场景，发现许多失败不仅是广泛的知识缺陷，也是局部的决策缺陷：重复猜测、格式错误的动作，以及违反模型刚刚看到的反馈。这些诊断促使我们设计了一个围绕三个步骤组织的训练方案：通过监督微调获得广泛的游戏参与能力，使用回合局部的偏好对修复目标对话游戏家族内可机械验证的失败，并保持这些对话游戏之外的一般能力。在官方最终评估中，我们的提交将公开clemscore从10.67提升至38.92，封闭域内得分从13.41提升至41.17，同时大致保持聚合静态性能不变（基线为44.14对44.24）。域外clemscore仍较低，为7.88，最大增益集中在目标家族未见过的变体上。我们的结果表明，广泛的SFT带来了模型能力提升的大部分；当失败检测精确时，回合局部监督可以发挥有效作用，观察到的迁移主要集中于家族内部。

English

Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.