展示一個例子，認識許多概念！在數學LLM中以反例驅動的概念推理

One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs

February 12, 2025

作者: Yinghui Li, Jiayi Kuang, Haojing Huang, Zhikun Xu, Xinnian Liang, Yi Yu, Wenlian Lu, Yangning Li, Xiaoyu Tan, Chao Qu, Ying Shen, Hai-Tao Zheng, Philip S. Yu

cs.AI

摘要

利用數學大型語言模型（LLMs）進行證明生成是LLMs研究中的基本主題。我們認為當前LLMs證明陳述的能力很大程度上取決於它們在訓練過程中是否遇到相應的證明過程。這種依賴限制了它們對數學定理及相關概念的深入理解。受到人類數學教育中常用的“反例證明”教學方法的啟發，我們的工作旨在通過反例來增強LLMs進行數學推理和證明的能力。具體而言，我們手動創建了一個高質量的大學水平數學基準CounterMATH，要求LLMs通過提供反例來證明數學陳述，從而評估它們對數學概念的掌握。此外，我們開發了一個數據工程框架，以自動獲取訓練數據以進一步改進模型。廣泛的實驗和詳細的分析表明CounterMATH具有挑戰性，表明像OpenAI o1這樣的LLMs在反例驅動的證明能力方面不足。此外，我們對模型訓練的探索顯示，加強LLMs的反例驅動概念推理能力對於提高它們的整體數學能力至關重要。我們相信我們的工作為數學LLMs社區提供了新的視角。

English

Leveraging mathematical Large Language Models (LLMs) for proof generation is a fundamental topic in LLMs research. We argue that the ability of current LLMs to prove statements largely depends on whether they have encountered the relevant proof process during training. This reliance limits their deeper understanding of mathematical theorems and related concepts. Inspired by the pedagogical method of "proof by counterexamples" commonly used in human mathematics education, our work aims to enhance LLMs' ability to conduct mathematical reasoning and proof through counterexamples. Specifically, we manually create a high-quality, university-level mathematical benchmark, CounterMATH, which requires LLMs to prove mathematical statements by providing counterexamples, thereby assessing their grasp of mathematical concepts. Additionally, we develop a data engineering framework to automatically obtain training data for further model improvement. Extensive experiments and detailed analyses demonstrate that CounterMATH is challenging, indicating that LLMs, such as OpenAI o1, have insufficient counterexample-driven proof capabilities. Moreover, our exploration into model training reveals that strengthening LLMs' counterexample-driven conceptual reasoning abilities is crucial for improving their overall mathematical capabilities. We believe that our work offers new perspectives on the community of mathematical LLMs.