LoopFormer：基於捷徑調變的潛在推理彈性深度循環Transformer

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation

February 11, 2026

作者: Ahmadreza Jeddi, Marco Ciccone, Babak Taati

cs.AI

摘要

迴圈式Transformer已成為語言領域中高效且強大的推理模型類別。近期研究表明，這類模型在算法與推理任務上表現卓越，顯示迴圈架構對潛在推理具有歸納偏置。然而，現有方法在訓練與推論階段固定了迴圈迭代次數，尚未解決這些模型能否在可變計算預算下靈活調整計算深度的問題。我們提出LoopFormer——一種基於可變長度軌跡訓練的迴圈式Transformer，可實現預算條件化推理。核心貢獻在於「捷徑一致性」訓練方案，該方案通過對齊不同長度的軌跡，確保短迴圈能產生具信息量的表徵，而長迴圈則持續優化之。LoopFormer使每個迴圈根據當前時間與步長進行條件化計算，讓表徵在不同長度軌跡間保持一致性演化，避免偏移或停滯。實證結果顯示，即便在嚴格計算限制下，LoopFormer於語言建模與推理基準測試中仍保持穩健性能，並能隨預算增加而優雅擴展。這些成果證實迴圈式Transformer本質上適合自適應語言建模，為可控且具預算感知能力的大型語言模型開闢了新路徑。

English

Looped Transformers have emerged as an efficient and powerful class of models for reasoning in the language domain. Recent studies show that these models achieve strong performance on algorithmic and reasoning tasks, suggesting that looped architectures possess an inductive bias toward latent reasoning. However, prior approaches fix the number of loop iterations during training and inference, leaving open the question of whether these models can flexibly adapt their computational depth under variable compute budgets. We introduce LoopFormer, a looped Transformer trained on variable-length trajectories to enable budget-conditioned reasoning. Our core contribution is a shortcut-consistency training scheme that aligns trajectories of different lengths, ensuring that shorter loops yield informative representations while longer loops continue to refine them. LoopFormer conditions each loop on the current time and step size, enabling representations to evolve consistently across trajectories of varying length rather than drifting or stagnating. Empirically, LoopFormer demonstrates robust performance on language modeling and reasoning benchmarks even under aggressive compute constraints, while scaling gracefully with additional budget. These results show that looped Transformers are inherently suited for adaptive language modeling, opening a path toward controllable and budget-aware large language models.