言语化采样：如何缓解模式崩溃并释放大语言模型的多样性潜力

摘要

后训练对齐往往降低大语言模型（LLM）的多样性，引发一种称为模式崩溃的现象。与以往研究将此效应归因于算法局限不同，我们揭示了一个根本且普遍存在的数据层面驱动因素：偏好数据中的典型性偏差，即标注者因认知心理学中已确立的发现而系统性地偏好熟悉文本。我们理论化这一偏差，在偏好数据集上实证验证，并展示其在模式崩溃中的核心作用。基于此分析，我们提出了“言语化采样”（Verbalized Sampling, VS），一种简单、无需训练的提示策略，以规避模式崩溃。VS提示模型对一组响应（如“生成5个关于咖啡的笑话及其对应概率”）进行概率分布的言语化表达。全面实验表明，VS在创意写作（诗歌、故事、笑话）、对话模拟、开放式问答及合成数据生成等方面显著提升性能，且不牺牲事实准确性与安全性。例如，在创意写作中，VS较直接提示将多样性提高了1.6至2.1倍。我们还观察到一种新兴趋势，即能力更强的模型从VS中获益更多。总之，我们的工作为模式崩溃提供了一个新的数据中心视角，以及一种实用的推理时补救措施，有助于释放预训练生成模型的多样性潜力。

English

Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling, a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., ``Generate 5 jokes about coffee and their corresponding probabilities''). Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

言语化采样：如何缓解模式崩溃并释放大语言模型的多样性潜力

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

摘要

Support