CoPE：截断式旋转位置编码——长上下文大语言模型的可扩展免费午餐

CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs

February 5, 2026

作者: Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang

cs.AI

摘要

旋转位置编码（RoPE）是大语言模型语境扩展的核心组件。尽管已有多种方法被提出用于使RoPE适应更长语境，其指导原则主要可归为两类：（1）分布外泛化，通过调整RoPE频率以适应未见位置；（2）语义建模，主张基于RoPE计算的注意力分数应始终优先关注语义相似的词元。本研究通过极简干预策略——即软截断RoPE低频分件的CoPE方法——统一了这两个看似独立的目标。CoPE不仅能消除分布外异常值并优化语义信号，还可避免硬截断引发的频谱泄漏。大量实验表明，仅对RoPE施加软截断策略即可获得显著的性能提升，其语境长度扩展能力最高达256k，验证了我们的理论分析，并使CoPE成为长度泛化领域的新标杆。相关代码、数据及模型已开源：https://github.com/hrlics/CoPE。

English

Rotary Positional Embedding (RoPE) is a key component of context scaling in Large Language Models (LLMs). While various methods have been proposed to adapt RoPE to longer contexts, their guiding principles generally fall into two categories: (1) out-of-distribution (OOD) mitigation, which scales RoPE frequencies to accommodate unseen positions, and (2) Semantic Modeling, which posits that the attention scores computed with RoPE should always prioritize semantically similar tokens. In this work, we unify these seemingly distinct objectives through a minimalist intervention, namely CoPE: soft clipping lowfrequency components of RoPE. CoPE not only eliminates OOD outliers and refines semantic signals, but also prevents spectral leakage caused by hard clipping. Extensive experiments demonstrate that simply applying our soft clipping strategy to RoPE yields significant performance gains that scale up to 256k context length, validating our theoretical analysis and establishing CoPE as a new state-of-the-art for length generalization. Our code, data, and models are available at https://github.com/hrlics/CoPE.