大型語言模型透過潛在蒸餾進行探索

Large Language Models Explore by Latent Distilling

April 27, 2026

作者: Yuanhao Zeng, Ao Lu, Lufei Li, Zheng Zhang, Yexin Li, Kan Ren

cs.AI

摘要

生成多樣化回應對於大型語言模型（LLM）的測試時擴展至關重要，然而標準的隨機取樣通常僅產生表層詞彙變化，限制了語義探索的深度。本文提出探索性取樣（ESamp），一種在生成過程中顯式鼓勵語義多樣性的解碼方法。ESamp的動機源於一個廣為人知的觀察：神經網絡對類似過往遭遇的輸入往往能做出低誤差預測，而對新穎輸入則會產生較高預測誤差。基於此特性，我們在測試時訓練一個輕量級蒸餾器，通過LLM的淺層表徵預測其深層隱藏表徵，從而建模LLM的深度方向表徵轉換。在解碼過程中，蒸餾器持續適應當前生成上下文誘導的映射關係。ESamp將預測誤差作為新穎性信號，對基於當前前綴的候選詞元擴展進行重新加權，從而將解碼偏向於未被充分探索的語義模式。ESamp採用異步訓練-推理流水線實現，最壞情況下開銷低於5%（優化版本為1.2%）。實證結果表明，ESamp能顯著提升推理模型的Pass@k效率，表現優於或媲美強力的隨機與啟發式基線。值得注意的是，ESamp在數學、科學和代碼生成基準測試中展現出穩健的泛化能力，並打破了創意寫作中多樣性與連貫性的權衡取捨。我們的程式碼已發佈於：https://github.com/LinesHogan/tLLM。

English

Generating diverse responses is crucial for test-time scaling of large language models (LLMs), yet standard stochastic sampling mostly yields surface-level lexical variation, limiting semantic exploration. In this paper, we propose Exploratory Sampling (ESamp), a decoding approach that explicitly encourages semantic diversity during generation. ESamp is motivated by the well-known observation that neural networks tend to make lower-error predictions on inputs similar to those encountered before, and incur higher prediction error on novel ones. Building on this property, we train a lightweight Distiller at test time to predict deep-layer hidden representations of the LLM from its shallow-layer representations to model the LLM's depth-wise representation transitions. During decoding, the Distiller continuously adapts to the mappings induced by the current generation context. ESamp uses the prediction error as a novelty signal to reweight candidate token extensions conditioned on the current prefix, thereby biasing decoding toward less-explored semantic patterns. ESamp is implemented with an asynchronous training--inference pipeline, with less than 5% worst case overhead (1.2% in the optimized release). Empirical results show that ESamp significantly boosts the Pass@k efficiency of reasoning models, showing superior or comparable performance to strong stochastic and heuristic baselines. Notably, ESamp achieves robust generalization across mathematics, science, and code generation benchmarks and breaks the trade-off between diversity and coherence in creative writing. Our code has released at: https://github.com/LinesHogan/tLLM.