言前先知：大语言模型表征在完成前即编码思维链成功信息

摘要

我们探究了零样本思维链（CoT）过程的成功是否能在完成前被预测。研究发现，基于大语言模型（LLM）表示的探测分类器，在生成首个词元前已表现出色，这表明推理过程中的关键信息已蕴含于初始步骤的表示之中。相比之下，仅依赖生成词元的强BERT基线模型表现较差，可能因其依赖于浅层语言线索而非深层推理动态。令人意外的是，使用后续推理步骤并不总能提升分类效果。当额外上下文无益时，早期表示与后期表示更为相似，暗示LLM在早期就已编码关键信息。这意味着推理过程往往可提前终止而不失效果。为验证此点，我们进行了早期停止实验，结果显示，即便截断CoT推理，其表现仍优于完全不使用CoT，尽管与完整推理相比仍存在差距。然而，旨在缩短CoT链的监督学习或强化学习方法，可借助我们分类器的指导来识别何时早期停止有效。我们的发现为支持此类方法提供了洞见，有助于在保持CoT优势的同时优化其效率。

English

We investigate whether the success of a zero-shot Chain-of-Thought (CoT) process can be predicted before completion. We discover that a probing classifier, based on LLM representations, performs well even before a single token is generated, suggesting that crucial information about the reasoning process is already present in the initial steps representations. In contrast, a strong BERT-based baseline, which relies solely on the generated tokens, performs worse, likely because it depends on shallow linguistic cues rather than deeper reasoning dynamics. Surprisingly, using later reasoning steps does not always improve classification. When additional context is unhelpful, earlier representations resemble later ones more, suggesting LLMs encode key information early. This implies reasoning can often stop early without loss. To test this, we conduct early stopping experiments, showing that truncating CoT reasoning still improves performance over not using CoT at all, though a gap remains compared to full reasoning. However, approaches like supervised learning or reinforcement learning designed to shorten CoT chains could leverage our classifier's guidance to identify when early stopping is effective. Our findings provide insights that may support such methods, helping to optimize CoT's efficiency while preserving its benefits.

言前先知：大语言模型表征在完成前即编码思维链成功信息

Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion

摘要

Support