變壓器為何無法在上下文中預測時間序列？

Why Do Transformers Fail to Forecast Time Series In-Context?

October 10, 2025

作者: Yufa Zhou, Yixiao Wang, Surbhi Goel, Anru R. Zhang

cs.AI

摘要

时间序列预测（TSF）在机器学习领域中依然是一个具有挑战性且尚未完全解决的问题，尽管近期大量研究利用大型语言模型（LLMs），这些模型主要依赖于Transformer架构。实证研究一致表明，即使在TSF任务中，强大的Transformer模型也往往无法超越更为简单的模型，例如线性模型；然而，对于这一现象的严格理论理解仍然有限。本文通过上下文学习（ICL）理论的视角，对Transformer在TSF中的局限性进行了理论分析。具体而言，在AR(p)数据下，我们确立了以下几点：（1）线性自注意力（LSA）模型在上下文预测中无法实现比经典线性模型更低的期望均方误差；（2）当上下文长度趋近于无穷大时，LSA渐近地恢复最优线性预测器；（3）在思维链（CoT）式推理下，预测值以指数速度收敛至均值。我们通过精心设计的实验对这些发现进行了实证验证。我们的理论不仅揭示了几个先前未被充分探讨的现象，还为设计更有效的预测架构提供了实用见解。我们希望我们的工作能够鼓励更广泛的研究社区重新审视TSF的基本理论局限性，并在未经深入审查的情况下，对日益复杂架构的直接应用进行批判性评估。

English

Time series forecasting (TSF) remains a challenging and largely unsolved problem in machine learning, despite significant recent efforts leveraging Large Language Models (LLMs), which predominantly rely on Transformer architectures. Empirical evidence consistently shows that even powerful Transformers often fail to outperform much simpler models, e.g., linear models, on TSF tasks; however, a rigorous theoretical understanding of this phenomenon remains limited. In this paper, we provide a theoretical analysis of Transformers' limitations for TSF through the lens of In-Context Learning (ICL) theory. Specifically, under AR(p) data, we establish that: (1) Linear Self-Attention (LSA) models cannot achieve lower expected MSE than classical linear models for in-context forecasting; (2) as the context length approaches to infinity, LSA asymptotically recovers the optimal linear predictor; and (3) under Chain-of-Thought (CoT) style inference, predictions collapse to the mean exponentially. We empirically validate these findings through carefully designed experiments. Our theory not only sheds light on several previously underexplored phenomena but also offers practical insights for designing more effective forecasting architectures. We hope our work encourages the broader research community to revisit the fundamental theoretical limitations of TSF and to critically evaluate the direct application of increasingly sophisticated architectures without deeper scrutiny.