DiagEvo：基於層次化錯誤記憶的診斷引導自我進化

摘要

自我對弈是語言模型自我進化的有效範式，但在缺乏引導的情況下，求解器的表現可能在後續回合中停滯甚至衰退。無引導方法利用難度、可學習性或多樣性等訊號來引導題目生成。這些訊號使題目具有挑戰性且多樣化，但並未指明後續回合應針對哪些未解決的推理弱點。引導方法則從外部任務資源取得方向，包括人類範例、文件語料庫或指定的難度目標，因此依賴於自我對弈迴圈之外所供給的任務資訊。我們證明，所需的方向可以改由求解器自身的失敗歷史推導而來。我們提出 DiagEvo，其中的診斷器從此歷史中萃取反覆出現的錯誤成因，並將其儲存在階層式錯誤成因記憶中。該記憶將相關成因歸類於技能節點之下，並根據目標題目上的自我一致性，將每個成因標記為「活躍」（Active）或「已掌握」（Mastered）。挑戰者利用這些狀態與重現次數，在針對成因的題目生成與自由探索之間取得平衡。雙重置信過濾僅在求解器最常見答案具有明顯票數領先時，才保留中等難度的題目。DiagEvo 從自我對弈過程中所產生的資訊推導其課程，而無需外部任務資源。在預設的 4B 診斷器之下，對於三個求解器——Qwen3-4B、Qwen3-8B 和 OctoThinker-8B——DiagEvo 在全部九個基準上的平均準確率均勝過所有基線。在 Qwen3-8B 上，它在五個數學推理基準上的平均準確率達 72.3%，比 R-Zero 高出 4.5 個百分點；其在全部九個基準上的平均準確率為 57.4%，比 DARC 高 1.1 個百分點。消融實驗顯示，階層式錯誤成因記憶與雙重置信過濾對這些提升皆有貢獻。

English

Self-play is an effective paradigm for language-model self-evolution, but without guidance, solver performance can plateau or decline across rounds. Unguided methods steer question generation with signals such as difficulty, learnability, or diversity. These signals keep questions challenging and varied but do not specify which unresolved reasoning weaknesses later rounds should target. Guided methods obtain direction from external task resources, including human examples, document corpora, or specified difficulty targets, and therefore rely on task information supplied outside the self-play loop. We show that the needed direction can instead be derived from the solver's own failure history. We introduce DiagEvo, whose diagnostician extracts recurring error causes from this history and stores them in a hierarchical error-cause memory. The memory groups related causes under skill nodes and tracks each as Active or Mastered according to self-consistency on targeted questions. The challenger uses these states and recurrence counts to balance cause-targeted generation with free exploration. Double-confidence filtering retains intermediate-difficulty questions only when the most common solver answer has a clear vote lead. DiagEvo derives its curriculum from information produced during self-play, without external task resources. With the default 4B diagnostician, DiagEvo outperforms every baseline in mean accuracy across all nine benchmarks for each of the three solvers: Qwen3-4B, Qwen3-8B, and OctoThinker-8B. On Qwen3-8B, it reaches 72.3% mean accuracy across five mathematical reasoning benchmarks, 4.5 percentage points above R-Zero. Its mean accuracy across all nine benchmarks is 57.4%, 1.1 percentage points above DARC. Ablations show that the hierarchical error-cause memory and double-confidence filtering both contribute to these gains.