LoopFormer：基于捷径调制的弹性深度循环Transformer潜空间推理框架（注：采用"潜空间推理"而非直译"潜在推理"以更准确体现latent reasoning的技术内涵；"弹性深度循环"准确传达elastic-depth looped特性；"捷径调制"精准对应shortcut modulation技术概念；整体采用符合中文论文标题规范的学术表达）

摘要

循环Transformer已成为语言推理领域一类高效且强大的模型。近期研究表明，这类模型在算法和推理任务上表现优异，暗示循环架构对潜在推理具有归纳偏置。然而，现有方法在训练和推理时固定循环迭代次数，尚未解决这些模型能否在可变计算预算下灵活调整计算深度的问题。我们提出LoopFormer——一种在可变长度轨迹上训练的循环Transformer，可实现预算条件化推理。我们的核心贡献是捷径一致性训练方案，该方案通过对齐不同长度的轨迹，确保短循环产生信息丰富的表征，而长循环持续优化这些表征。LoopFormer使每个循环基于当前时间和步长进行条件化，让表征能在不同长度轨迹间保持一致性演化，而非漂移或停滞。实验表明，即使在严格的计算限制下，LoopFormer在语言建模和推理基准测试中仍保持稳健性能，并能随预算增加优雅扩展。这些结果证明循环Transformer天生适合自适应语言建模，为开发可控且具备预算意识的大型语言模型开辟了新路径。

English

Looped Transformers have emerged as an efficient and powerful class of models for reasoning in the language domain. Recent studies show that these models achieve strong performance on algorithmic and reasoning tasks, suggesting that looped architectures possess an inductive bias toward latent reasoning. However, prior approaches fix the number of loop iterations during training and inference, leaving open the question of whether these models can flexibly adapt their computational depth under variable compute budgets. We introduce LoopFormer, a looped Transformer trained on variable-length trajectories to enable budget-conditioned reasoning. Our core contribution is a shortcut-consistency training scheme that aligns trajectories of different lengths, ensuring that shorter loops yield informative representations while longer loops continue to refine them. LoopFormer conditions each loop on the current time and step size, enabling representations to evolve consistently across trajectories of varying length rather than drifting or stagnating. Empirically, LoopFormer demonstrates robust performance on language modeling and reasoning benchmarks even under aggressive compute constraints, while scaling gracefully with additional budget. These results show that looped Transformers are inherently suited for adaptive language modeling, opening a path toward controllable and budget-aware large language models.

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation

摘要

Support