A2RBench：一種用於可形式驗證抽象推理基準生成的自動化範式

摘要

抽象推理能力反映了大型語言模型（LLM）提取並應用抽象規則的智慧與泛化能力。然而，準確衡量此能力仍具挑戰：現有基準測試若非依賴昂貴的人工標註（限制其規模），便可能測量到的是記憶而非真正的推理。為解決此問題，我們提出名為A2RBench的自動化流程，涵蓋生成、擴展、評估與分析。具體而言，在生成階段，LLM建立需真正推理的多樣化任務；在擴展階段，LLM重複使用已驗證規則並擴展新輸入空間以產生任務變體，實現規模化。然而，此過程可能引發幻覺。為消除此問題，我們進一步建立理論框架，證明程式化驗證——測試逆向操作是否能完美還原正向操作（循環一致性）——可保證唯一解。透過對主流LLM的廣泛評估，我們發現：（1）當前LLM在抽象推理上存在根本性缺陷，頂尖模型在代表性子集上表現顯著不如人類（39.8% vs. 68.5%）。（2）當前LLM在生成的三維任務複雜度上遠不及二維與一維任務，揭示其對高維任務的理解不足。（3）違反直覺的是，資訊複雜度較高的輸入反而能簡化推理過程。

English

Abstract reasoning ability reflects the intelligence and generalization capacity of LLMs to extract and apply abstract rules. However, accurately measuring this ability remains challenging: existing benchmarks either rely on expensive manual annotation, limiting their scale, or risk measuring memorization rather than genuine reasoning. To address this, we introduce an automated pipeline named A2RBench, encompassing generation, expansion, evaluation, and analysis. Specifically, in the generation stage, LLMs create diverse tasks demanding genuine reasoning; in the expansion stage, LLMs reuse validated rules and expand new input spaces to generate task variations, achieving scaling. However, such a process may cause hallucinations. To eliminate it, we further establish a theoretical framework and prove that programmatic verification--testing whether the inverse operation perfectly reverses the forward operation (cycle consistency)--guarantees a unique solution. Through extensive evaluations on mainstream LLMs, we find: (1) Current LLMs exhibit fundamental deficiencies in abstract reasoning, with top models significantly underperforming humans on a representative subset (39.8% vs. 68.5%). (2) Current LLMs fall far short of 2D and 1D in the complexity of generated 3D tasks, revealing their lack of understanding of high-dimensional tasks. (3) Counterintuitively, inputs with higher information complexity can simplify the reasoning process.