CRISP：基于结构质量驱动的路由的悬崖感知输入自适应稀疏预填充

摘要

长上下文大语言模型推理中，注意力预填充阶段的开销随序列长度呈二次方增长，使自注意力成为严重的计算瓶颈。传统稀疏注意力方法通过固定模式或离线分析来缓解这一问题，但缺乏对输入相关注意力结构的自适应灵活性。近年来，动态方法通过实时将注意力头路由至稀疏模式来应对这一挑战，但其依赖开销较大的间接路由代理，且其预算分配机制忽略了softmax后的质量层级结构。我们提出CRISP（悬崖感知的输入自适应稀疏预填充），识别并解决了该动态路由范式中的两个结构性挑战。首先，我们证明路由决策可以直接从代理注意力图的结构中读取。我们以结构代理C_struct取代基于Jensen-Shannon散度（JSD）的路由策略；C_struct在垂直-斜线兼容位置上度量质量分布，能够在复现JSD路由决策的同时，消除池化矩阵乘法及其后KL散度的开销。其次，我们对softmax后的质量悬崖进行了形式化刻画，并从理论上证明：在长上下文中，严格累积覆盖阈值会累积O(n)的背景噪声。CRISP通过一种基于噪声基底的汇感知阈值来规避这一问题。实验方面，在InfiniteBench、RULER和LongBench上对两个模型家族的评估显示，CRISP总体上是性能最强的稀疏方法，在检索密集型基准上达到甚至超越精确密集注意力；其检索任务表现相较基线最高提升28.0个百分点，在512k token下实现最高5.30倍的注意力加速——这主要得益于我们在选择过程中消除O(n)噪声的同时保持了结构完整性。

English

The attention prefilling phase of long-context LLM inference scales quadratically, making self-attention a severe computational bottleneck. Traditional sparse attention methods mitigate this through fixed patterns or offline profiling, but lack the flexibility to adapt to input-dependent attention structure. Recent dynamic methods address this by routing heads to sparse patterns in real-time, but rely on indirect routing proxies with overhead and budget allocation mechanisms that overlook the post-softmax mass hierarchy. We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm. First, we show that the routing decision can be read directly off the structure of the proxy attention map. We replace the Jensen-Shannon Divergence (JSD) routing with C_struct, a structural proxy that measures mass at Vertical-Slash compatible positions and reproduces JSD's routing decisions while eliminating both the pooled matmul and subsequent KL divergence overhead. Second, we formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts. CRISP navigates this via a sink-aware threshold grounded in the noise floor. Empirically, across InfiniteBench, RULER and LongBench on two model families, CRISP is the strongest sparse method overall and matches or exceeds exact dense attention on retrieval-heavy benchmarks, recovering up to +28.0 pp on retrieval tasks over baselines and achieving up to a 5.30x attention speedup at 512k tokens, driven primarily by our O(n) noise elimination during selection while preserving structural integrity.