口語化抽樣：如何緩解模式崩潰並釋放大語言模型的多樣性

摘要

訓練後對齊往往會降低大型語言模型（LLM）的多樣性，導致一種被稱為模式崩潰的現象。與先前研究將此效應歸因於算法限制不同，我們發現了一個根本且普遍存在於數據層面的驅動因素：偏好數據中的典型性偏見，即由於認知心理學中已確立的發現，註釋者系統性地偏好熟悉的文本。我們從理論上形式化了這種偏見，並在偏好數據集上進行了實證驗證，顯示其在模式崩潰中扮演了核心角色。基於此分析，我們引入了「言語化抽樣」（Verbalized Sampling, VS），這是一種簡單、無需訓練的提示策略，旨在規避模式崩潰。VS提示模型對一組回應進行概率分佈的言語化描述（例如，「生成5個關於咖啡的笑話及其對應的概率」）。全面的實驗表明，VS在創意寫作（詩歌、故事、笑話）、對話模擬、開放式問答以及合成數據生成等方面顯著提升了性能，且未犧牲事實準確性和安全性。例如，在創意寫作中，VS相比直接提示將多樣性提高了1.6至2.1倍。我們進一步觀察到一個新興趨勢，即能力更強的模型從VS中獲益更多。總之，我們的工作為模式崩潰提供了一個新的以數據為中心的視角，以及一個實用的推理時補救措施，有助於釋放預訓練生成模型的多樣性潛力。

English

Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we identify a fundamental, pervasive data-level driver: typicality bias in preference data, whereby annotators systematically favor familiar text as a result of well-established findings in cognitive psychology. We formalize this bias theoretically, verify it on preference datasets empirically, and show that it plays a central role in mode collapse. Motivated by this analysis, we introduce Verbalized Sampling, a simple, training-free prompting strategy to circumvent mode collapse. VS prompts the model to verbalize a probability distribution over a set of responses (e.g., ``Generate 5 jokes about coffee and their corresponding probabilities''). Comprehensive experiments show that VS significantly improves performance across creative writing (poems, stories, jokes), dialogue simulation, open-ended QA, and synthetic data generation, without sacrificing factual accuracy and safety. For instance, in creative writing, VS increases diversity by 1.6-2.1x over direct prompting. We further observe an emergent trend that more capable models benefit more from VS. In sum, our work provides a new data-centric perspective on mode collapse and a practical inference-time remedy that helps unlock pre-trained generative diversity.

口語化抽樣：如何緩解模式崩潰並釋放大語言模型的多樣性

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

摘要

Support