CRISP：懸崖感知、輸入自適應的稀疏預填充，採用結構質量驅動的路由

摘要

長上下文 LLM 推論的注意力預填充階段呈二次方擴展，使自注意力成為嚴重的計算瓶頸。傳統的稀疏注意力方法透過固定模式或離線剖析來緩解此問題，但缺乏適應輸入相關注意力結構的靈活性。近期的動態方法透過即時將注意力頭路由至稀疏模式來解決此問題，然而此類方法依賴於具有額外開銷的間接路由代理，且其預算分配機制忽視了 softmax 後的質量層級結構。我們提出 CRISP（懸崖感知輸入自適應稀疏預填充，Cliff-awaRe Input-adaptive Sparse Prefilling），識別並解決了此動態路由範式中的兩個結構性挑戰。首先，我們證明路由決策可以直接從代理注意力圖的結構中讀取。我們以 C_struct 取代詹森-夏農散度（JSD）路由；C_struct 是一種結構性代理，用於量測 Vertical-Slash 相容位置上的質量分布，既能重現 JSD 的路由決策，同時消除池化矩陣乘法及後續 KL 散度的開銷。其次，我們將 softmax 後的質量懸崖形式化，並在理論上證明嚴格的累積覆蓋閾值會在長上下文中累積 O(n) 的背景雜訊。CRISP 透過以雜訊底為基礎的匯點感知閾值來應對此問題。在實驗中，涵蓋 InfiniteBench、RULER 與 LongBench 三個基準、橫跨兩個模型家族，CRISP 整體上是表現最強的稀疏方法，且在檢索密集型基準上達到或超越精確稠密注意力的表現，於檢索任務上相較基線提升高達 +28.0 個百分點，並在 512k tokens 下實現高達 5.30 倍的注意力加速。此成效主要歸因於我們在選擇過程中消除 O(n) 雜訊的同時保持了結構完整性。

English

The attention prefilling phase of long-context LLM inference scales quadratically, making self-attention a severe computational bottleneck. Traditional sparse attention methods mitigate this through fixed patterns or offline profiling, but lack the flexibility to adapt to input-dependent attention structure. Recent dynamic methods address this by routing heads to sparse patterns in real-time, but rely on indirect routing proxies with overhead and budget allocation mechanisms that overlook the post-softmax mass hierarchy. We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm. First, we show that the routing decision can be read directly off the structure of the proxy attention map. We replace the Jensen-Shannon Divergence (JSD) routing with C_struct, a structural proxy that measures mass at Vertical-Slash compatible positions and reproduces JSD's routing decisions while eliminating both the pooled matmul and subsequent KL divergence overhead. Second, we formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts. CRISP navigates this via a sink-aware threshold grounded in the noise floor. Empirically, across InfiniteBench, RULER and LongBench on two model families, CRISP is the strongest sparse method overall and matches or exceeds exact dense attention on retrieval-heavy benchmarks, recovering up to +28.0 pp on retrieval tasks over baselines and achieving up to a 5.30x attention speedup at 512k tokens, driven primarily by our O(n) noise elimination during selection while preserving structural integrity.