稜鏡：頻譜感知區塊稀疏注意力機制（注：標題採用技術術喻的翻譯策略，"Prism"直譯為"稜鏡"以保留光學隱喻，"Spectral-Aware"結合頻譜理論與感知機制譯為"頻譜感知"，"Block-Sparse Attention"則採用深度學習領域通用的"區塊稀疏注意力"譯法，整體突出該注意力機制兼具光譜分析特性與計算效率的雙重優勢。）

摘要

區塊稀疏注意力機制在加速長文本LLM預填充階段展現潛力，但如何高效識別相關區塊仍是瓶頸。現有方法通常採用粗粒度注意力作為區塊重要性評估的代理指標，卻往往依賴昂貴的詞元級搜索或評分機制，導致顯著的選擇開銷。本文通過理論分析指出，標準基於均值池化的粗粒度注意力不準確性根源在於：均值池化與旋轉位置編碼（RoPE）的交互作用。我們證明均值池化相當於低通濾波器，會在高頻維度引發破壞性干涉，從而對局部位置信息（如斜線模式）形成「盲區」。為解決此問題，我們提出無需訓練的頻譜感知方法Prism，將區塊選擇分解為高頻與低頻雙分支。通過基於能量的溫度校準技術，Prism能直接從池化表徵中恢復衰減的位置信號，實現純區塊級操作的區塊重要性評估，從而提升效率。大量實驗驗證表明，Prism在保持與全注意力機制精度相當的同時，可實現最高5.1倍的加速比。

English

Block-sparse attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically employ coarse-grained attention as a proxy for block importance estimation, but often resort to expensive token-level searching or scoring, resulting in significant selection overhead. In this work, we trace the inaccuracy of standard coarse-grained attention via mean pooling to a theoretical root cause: the interaction between mean pooling and Rotary Positional Embeddings (RoPE). We prove that mean pooling acts as a low-pass filter that induces destructive interference in high-frequency dimensions, effectively creating a "blind spot" for local positional information (e.g., slash patterns). To address this, we introduce Prism, a training-free spectral-aware approach that decomposes block selection into high-frequency and low-frequency branches. By applying energy-based temperature calibration, Prism restores the attenuated positional signals directly from pooled representations, enabling block importance estimation using purely block-level operations, thereby improving efficiency. Extensive evaluations confirm that Prism maintains accuracy parity with full attention while delivering up to 5.1times speedup.

Prism: Spectral-Aware Block-Sparse Attention

摘要

Support