層次蛋糕：大型語言模型層級內的令牌感知對比解碼

摘要

大型語言模型（LLMs）在自然語言理解與生成方面表現卓越，但在處理事實性錯誤方面仍顯脆弱，這限制了其在知識密集型任務中的可靠性。儘管解碼時策略提供了一種無需訓練的高效解決方案，現有方法通常將詞元層級與層級信號孤立處理，忽視了它們之間的聯合動態。本研究引入了一種詞元感知、層級定位的對比解碼方法，該方法將特定類型的詞元與其最具影響力的變壓器層對齊，以提升事實生成能力。通過實證注意力分析，我們識別出兩個關鍵模式：標點符號詞元在早期層級中佔據主導注意力，而概念詞元則在中間層級中主導語義推理。通過在相應深度選擇性地抑制對這些詞元類型的注意力，我們實現了受控事實退化的誘導，並提取出對比信號以指導最終的事實解碼。我們的方法無需額外訓練或模型修改，實驗結果表明，該方法在多個LLMs及多種基準測試中均能持續提升事實準確性。

English

Large language models (LLMs) excel at natural language understanding and generation but remain vulnerable to factual errors, limiting their reliability in knowledge-intensive tasks. While decoding-time strategies provide a promising efficient solution without training, existing methods typically treat token-level and layer-level signals in isolation, overlooking the joint dynamics between them. In this work, we introduce a token-aware, layer-localized contrastive decoding method that aligns specific token types with their most influential transformer layers to improve factual generation. Through empirical attention analysis, we identify two key patterns: punctuation tokens receive dominant attention in early layers, while conceptual tokens govern semantic reasoning in intermediate layers. By selectively suppressing attention to these token types at their respective depths, we achieve the induction of controlled factual degradation and derive contrastive signals to guide the final factual decoding. Our method requires no additional training or model modification, and experiments demonstrate that our method consistently improves factuality across multiple LLMs and various benchmarks.

層次蛋糕：大型語言模型層級內的令牌感知對比解碼

LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers

摘要

Support