超越序列順序：用於Transformer的語法感知位置嵌入

摘要

Transformer 中的位置嵌入（PE）會編碼詞元的距離與順序，但大多不考慮句法結構。我們提出語法感知位置嵌入（SiPE）：在預訓練期間，它從依存句法分析結果中學習輕量級句法先驗，並將此先驗注入所有三種主流 PE 家族（絕對、相對、旋轉），適用於編碼器與解碼器，同時不更動自注意力與架構其餘部分。我們分別檢驗了先驗應在何處、以何種方式進入模型，發現這取決於架構：對使用相對 PE 的自回歸解碼器而言，先驗與注意力分數中的相對位置項進行乘法耦合時效果最強，優於將先驗注入輸入嵌入、注入自注意力，或同時注入位置項與注意力項的做法；對編碼器而言，則最好直接加至輸入嵌入，與各編碼器原生的位置機制相互配合。我們發現，使用 SiPE 預訓練的模型在 SyntaxGym 基準上取得高達 10.3% 的提升，同時相對於沒有句法監督的基準模型，困惑度降低 9.0%——這是一項幾乎所有現有句法注入方法反而會使之惡化的指標。關鍵的是，這些收益不限於句法泛化：SiPE 也改善了真實世界的語言理解，並使 GLUE 基準上的分數較未使用 SiPE 訓練的模型提升高達 8.2%。不同於現有句法語言模型在推論時對多個句法分析結果進行邊緣化，或在執行階段捨棄句法資訊，SiPE 僅以單一句法分析結果為條件，從而在句法監督與推論成本之間建立了新的 Pareto 前緣。

English

Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to syntactic structure. We introduce Syntax-informed Positional Embeddings (SiPE), which learns a lightweight syntactic prior from dependency parses during pretraining and injects it across all three dominant PE families (absolute, relative, rotary), for both encoders and decoders, leaving self-attention and the rest of the architecture untouched. We isolate where and how the prior should enter the model, and find it depends on the architecture: for autoregressive decoders that use relative PE, the prior is strongest when coupled multiplicatively with the relative-position term of the attention score, outperforming injection into the input embeddings, into self-attention, or into the positional and attention terms jointly---while for encoders it is best added directly to the input embeddings, composing with each encoder's native positional mechanism. We find that models pre-trained with SiPE improve on the SyntaxGym benchmark by up to 10.3% while simultaneously reducing perplexity by 9.0% over a base model with no syntactic supervision---a metric nearly every existing syntax-injection method instead degrades. Crucially, these gains extend beyond syntactic generalization: SiPE also improves real-world language understanding, raising scores on the GLUE benchmark by up to 8.2% over a model trained without it. Unlike existing syntactic language models that marginalize over many parses at inference or discard syntax at runtime, SiPE conditions on a single parse, establishing a new Pareto frontier between syntactic supervision and inference cost.