機器之實用心智：探尋大型語言模型中實用能力的湧現

摘要

当前的大型语言模型（LLMs）在社会智能任务中展现出了新兴的能力，包括隐含意义解析（Sravanthi等人，2024）和心理理论推理（Shapira等人，2024），这两者均需深厚的语用理解。然而，LLMs在训练过程中如何获得这一能力仍鲜为人知。在本研究中，我们引入了ALTPRAG，一个基于替代语用概念的数据集，旨在评估不同训练阶段的LLMs能否准确推断出细微的说话者意图。每个实例配对两个语境适宜但语用上截然不同的续写，从而实现对语用解释和对比推理的精细评估。我们系统地评估了22个LLMs在关键训练阶段的表现：预训练、监督微调（SFT）和偏好优化，以探究语用能力的发展。我们的结果表明，即便是基础模型也对语用线索表现出显著的敏感性，且随着模型和数据规模的增加，这种敏感性持续提升。此外，SFT和RLHF进一步促进了能力的提升，特别是在认知语用推理方面。这些发现强调了语用能力作为LLM训练中涌现且组合性质的特征，并为模型与人类交际规范的对齐提供了新的见解。

English

Current large language models (LLMs) have demonstrated emerging capabilities in social intelligence tasks, including implicature resolution (Sravanthi et al. (2024)) and theory-of-mind reasoning (Shapira et al. (2024)), both of which require substantial pragmatic understanding. However, how LLMs acquire this competence throughout the training process remains poorly understood. In this work, we introduce ALTPRAG, a dataset grounded in the pragmatic concept of alternatives, designed to evaluate whether LLMs at different training stages can accurately infer nuanced speaker intentions. Each instance pairs two contextually appropriate but pragmatically distinct continuations, enabling fine-grained assessment of both pragmatic interpretation and contrastive reasoning. We systematically evaluate 22 LLMs across key training stages: pre-training, supervised fine-tuning (SFT), and preference optimization, to examine the development of pragmatic competence. Our results show that even base models exhibit notable sensitivity to pragmatic cues, which improves consistently with increases in model and data scale. Additionally, SFT and RLHF contribute further gains, particularly in cognitive-pragmatic reasoning. These findings highlight pragmatic competence as an emergent and compositional property of LLM training and offer new insights for aligning models with human communicative norms.

機器之實用心智：探尋大型語言模型中實用能力的湧現

The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models

摘要

Summary

Support

Support