让我们逐句进行预测

Let's Predict Sentence by Sentence

May 28, 2025

作者: Hyeonbin Hwang, Byeongguk Jeon, Seungone Kim, Jiyeon Kim, Hoyeon Chang, Sohee Yang, Seungpil Won, Dohaeng Lee, Youbin Ahn, Minjoon Seo

cs.AI

摘要

自回归语言模型（LMs）一次生成一个词元，而人类的推理则基于更高层次的抽象——句子、命题和概念。这种对比引发了一个核心问题：LMs能否同样学会在结构化的语义单元而非原始词元序列上进行推理？在本研究中，我们探讨了预训练LMs是否能够通过其已学习的表征被提升至此类抽象推理空间。我们提出了一种框架，该框架通过自回归预测下一句的连续嵌入，将预训练的词元级LM适配到句子空间操作。我们探索了两种受经典表示学习启发的嵌入范式：1）语义嵌入，通过自编码学习以保留表层意义；2）上下文嵌入，通过下一句预测训练以编码预期结构。我们在两种推理机制下评估这两种嵌入：离散化推理，在重新编码前将每个预测嵌入解码为文本；以及连续推理，完全在嵌入空间中进行推理以提高效率。在数学、逻辑、常识和规划四个领域中，连续推理下的上下文嵌入与思维链（CoT）相比表现出竞争力，同时平均减少了一半的推理时浮点运算次数（FLOPs）。我们还展示了可扩展性和模块化适应的早期迹象。最后，为了可视化潜在轨迹，我们引入了SentenceLens，一种将中间模型状态解码为可解释句子的诊断工具。综合来看，我们的结果表明，预训练LMs能够在潜在嵌入空间内有效过渡到抽象、结构化的推理。

English

Autoregressive language models (LMs) generate one token at a time, yet human reasoning operates over higher-level abstractions - sentences, propositions, and concepts. This contrast raises a central question- Can LMs likewise learn to reason over structured semantic units rather than raw token sequences? In this work, we investigate whether pretrained LMs can be lifted into such abstract reasoning spaces by building on their learned representations. We present a framework that adapts a pretrained token-level LM to operate in sentence space by autoregressively predicting continuous embeddings of next sentences. We explore two embedding paradigms inspired by classical representation learning: 1) semantic embeddings, learned via autoencoding to preserve surface meaning; and 2) contextual embeddings, trained via next-sentence prediction to encode anticipatory structure. We evaluate both under two inference regimes: Discretized, which decodes each predicted embedding into text before re-encoding; and Continuous, which reasons entirely in embedding space for improved efficiency. Across four domains - mathematics, logic, commonsense, and planning - contextual embeddings under continuous inference show competitive performance with Chain-of-Thought (CoT) while reducing inference-time FLOPs on average by half. We also present early signs of scalability and modular adaptation. Finally, to visualize latent trajectories, we introduce SentenceLens, a diagnostic tool that decodes intermediate model states into interpretable sentences. Together, our results indicate that pretrained LMs can effectively transition to abstract, structured reasoning within latent embedding spaces.