当注意力失明：ALiBi位置编码中的数值失效

摘要

我们识别出ALiBi位置编码一个此前被忽视的失败模式：其线性偏差缩放会导致浮点精度下溢，从而将很大一部分注意力权重置零，并使受影响的注意力头部分失明。我们分析了这一失败模式，描述了其影响，并考察了四种缓解策略。我们进一步证明，该失败模式也出现在基于ALiBi的最先进预训练模型中。使用1.48亿参数解码器模型进行的全面预训练实验，帮助我们将其影响与上下文外退化区分开来。我们发现，ALiBi的失败模式会严重损害词元检索，而对标准解码器基准只有轻微影响。我们提出了四种训练时缓解策略，并对其进行了单独及组合评估，发现对数缩放距离在passkey检索中带来最一致的改进。尽管存在这一问题，默认的ALiBi斜率仍然是出人意料的强基线，尤其是在大海捞针式检索中。基于这些发现，我们为使用ALiBi训练模型提供了具体建议。

English

We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi.