StudyBench：自我演化能否從教科書中榨取奧林匹亞競賽能力？

摘要

人類只需研讀少數幾本寫得好的教科書，就能精通一門學科並嘗試解決其最難的問題。我們主張，理想的自我進化方法應具備相同性質，也就是能自主從原始訓練材料中學習，以獲得可遷移的解題能力。然而，我們仍缺乏對其的直接量測。我們提出 StudyBench，一個受控的物理基準測試，直接量測自我進化方法將訓練材料轉換為能力的效率。我們將測試集組織為應用集與遷移集：應用集由困難的教科書問題組成，用以評估吸收能力；遷移集由奧林匹亞層級問題組成，用以評估遷移能力。我們在三個基礎模型上對具代表性的自我進化方法進行基準測試，發現應用集上的改進很少能轉移到更難的遷移集。一項引導消融實驗揭示出引導落差：即使是最強的方法，也只能彌補相同材料以情境內引導方式提供時所能解鎖之效能提升的一小部分。此外，每種方法都會遇到計算高原，在遠未耗盡其計算預算之前便已飽和。因此，剩餘落差是方法問題，而非資料或計算問題。透過提供乾淨且受控的基準測試，StudyBench 將自我進化的進展從開放式探索轉變為未來研究可衡量的目標。我們的程式碼已發布於 https://github.com/thunlp/StudyBench。

English

Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolution method should share the same property, that is autonomously learning from raw training material for transferable problem-solving capability. However, we still lack a direct measurement for it. We introduce StudyBench, a controlled physics benchmark that directly measures how efficiently a self-evolution method converts training material into capability. We organise the test set into an Application Set, consisting of difficult textbook problems and evaluating absorption ability, and a Transfer Set, consisting of olympiad-level problems and evaluating transfer ability. Benchmarking representative self-evolution methods across three base models, we find that improvements on the Application Set rarely translate to the harder Transfer Set. A guidance ablation exposes a Guidance Gap: even the strongest method closes only a small fraction of what the same material unlocks when supplied as in-context guidance. Besides, every method hits a Compute Plateau, saturating well before exhausting its compute budget. The remaining gap is therefore a method problem rather than a data or compute problem. By offering a clean and controlled benchmark, StudyBench turns self-evolution progress from an open-ended pursuit into a measurable target for future research. Our code is released at https://github.com/thunlp/StudyBench.