大型語言模型推理軌跡中的認知片段使可解釋的人類題目難度預測得以實現

摘要

預測人類題目難度是教育評量的核心環節，可靠的難度估計有助於確保公平性與有效構建測驗。現有方法多仰賴昂貴的人工校準或題目文本表徵，且僅能提供有限的認知歷程證據來說明題目為何困難。我們主張難度不僅應視為題目文本的屬性，更應是題目引發解題負荷時可觀測的結果。大型推理模型（LRM）能透過推理軌跡提供可擴展的歷程證據，但此類證據須經結構化處理，方能支援可解釋的建模。為此，我們提出「Epi2Diff」（Episode to Difficulty）架構，將LRM推理軌跡映射至具認知基礎的階段序列。這些階段將軌跡片段分組為功能性的解題狀態，使難度能透過推理規模、努力分配與狀態轉換來建模。Epi2Diff萃取緊湊的階段動態特徵，並將其與語義題目表徵結合，用於預測人類題目難度。在四個真實世界的人類難度資料集實驗中，Epi2Diff持續優於強基線方法，包括微調的小型語言模型、LLM上下文學習與監督式LLM適應。在源自SAT的分類基準測試中，Epi2Diff比監督式LLM微調基線平均相對提升8.1%。進一步分析顯示，較難的題目會引發更耗費心力、更反覆且更偏重實作的階段動態，而不僅是更長的回應。這些結果證明，LRM推理軌跡中的認知階段能為人類題目難度提供具預測性與可解釋性的歷程表徵，為以推理模型進行教育評量開闢新視角。

English

Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.