我們衡量的是策略還是措辭？大型語言模型數學推理中表面層級與方法層級多樣性之間的鴻溝

摘要

LLM在數學推理中的多樣性對於探索至關重要，然而常見的多樣性指標大多僅捕捉表面層級的變化，而非問題解決方式上的差異。我們透過引入「策略層級多樣性」——即針對同一問題的正確解法中，所採用的解題策略差異——來填補此一缺口。運用經人類校準的LLM評判框架，我們發現先前的多樣性衡量指標無法可靠地反映策略層級多樣性，而此一落差亦延續至具多樣性感知的RLVR中：當目標指標得以保留時，策略層級多樣性反而下降。在探討策略層級多樣性何時有助益，以及能否直接誘發此類多樣性時，我們發現策略多樣的候選集合能改善測試階段的擴展效能。然而，在訓練過程中優化LLM評判的多樣性獎勵，會導致策略模型利用評判特有的偏好，而非拓展其解題策略，使得直接優化策略層級多樣性仍是一項未解難題。綜合而言，本研究提出策略層級多樣性的概念，並揭示表面層級與策略層級信號之間的系統性分歧，為實現LLM以真正多元且貼近人類的方式進行推理，邁出重要一步。

English

Diversity in LLM mathematical reasoning is critical for exploration, but common diversity metrics mostly capture surface-level variation rather than differences in how a problem is solved. We address this gap by introducing approach-level diversity: variation in strategies across correct solutions to the same problem. Using a human-calibrated LLM judge framework, we show that prior diversity measures are unreliable proxies for approach-level diversity, and this mismatch carries over to diversity-aware RLVR, where target metrics are preserved while approach-level diversity declines. Investigating when approach-level diversity helps and whether it can be directly induced, we find that approach-diverse candidate sets improve test-time scaling. However, optimizing an LLM judge diversity reward during training causes the policy to exploit judge-specific preferences rather than broaden its approaches, leaving direct optimization of approach-level diversity as an open problem. Together, our work introduces the notion of approach-level diversity and uncovers a systematic divergence between surface- and approach-level signals, marking a step toward LLMs that reason in genuinely diverse, human-like ways.