獲取、修復、保留：針對小模型對話遊戲智能體的診斷引導後訓練方法

摘要

互動式對話遊戲所考驗的能力，大多在靜態基準中未被明言：模型必須在回合間維護狀態、詮釋回饋，並在不斷變化的約束下選擇有效動作。我們在 LM Playschool 挑戰賽中，以一個 20 億參數的開放權重模型研究此一情境，發現許多失敗不僅源自廣泛的知識缺漏，也來自局部的決策失誤：重複猜測、格式錯誤的動作，以及違反模型剛才才看到的回饋。這些診斷結果促成了一套以三個步驟為架構的訓練方案：首先透過監督式微調獲取廣泛的遊戲參與能力；其次在單一目標對話遊戲家族內，利用回合局部的偏好對修復可機械驗證的失敗；最後則是在這些對話遊戲之外，保留通用能力。在官方最終評測中，我們的提交將公開 clemscore 從 10.67 提升至 38.92，封閉域內分數從 13.41 提升至 41.17，同時大致保留了靜態整體表現（基準為 44.14 對 44.24）。域外 clemscore 仍偏低，僅為 7.88，最大幅度的提升集中於目標家族未見過的變體。我们的结果提示，廣泛的 SFT 帶來了模型能力提升的大部分效益；當失敗偵測足夠精確時，回合局部的監督可以發揮效用，而觀察到的遷移主要集中於家族內部。

English

Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.