Prism-Δ：面向大语言模型提示凸显的差分子空间引导方法

摘要

提示高亮技术能够引导大语言模型在生成过程中优先处理用户指定的文本片段。其核心挑战在于提取能够区分相关与无关语境的引导方向，而非二者共有的结构模式。我们提出PRISM-Δ（基于投影的相关性感知引导方法），该方法通过分解正负交叉协方差矩阵的差异，在消除共享方向的同时最大化判别能量。每个注意力头会获得连续的softplus重要性权重，使得弱效但有用的注意力头能以降低的强度参与计算。该框架可自然扩展至Value表示，捕获仅使用Key的方法所忽略的内容通道信号。在四个基准测试和五个模型上的实验表明，PRISM-Δ在20种配置中的19种达到或超越现有最佳方法，相对增益最高达+10.6%，同时将引导的流畅性损耗降低一半。PRISM-Δ还能扩展至长上下文检索任务，相较现有最佳方法实现最高+4.8%的相对增益。该方法兼容FlashAttention且仅增加可忽略的内存开销。

English

Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-Δ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, PRISM-Δ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the fluency cost of steering. PRISM-Δ also scales to long-context retrieval, outperforming the best existing method by up to +4.8% relative gain. PRISM-Δ is compatible with FlashAttention and adds negligible memory overhead.

Prism-Δ：面向大语言模型提示凸显的差分子空间引导方法

Prism-Δ: Differential Subspace Steering for Prompt Highlighting in Large Language Models

摘要

Support