解码层禁忌：LLM鲁棒性的诊断性压力测试

摘要

大语言模型评估通常聚焦于标称条件下的性能，这制造了一种能力错觉，仿佛模型能够轻松行走在一条狭窄且高度优化的生成走廊上。然而，在现实部署中，复杂的系统提示、安全护栏和结构性约束不断迫使模型偏离这一标称路径，导致基准分数与部署性能之间出现偏差。为解决这一问题，我们引入了“解码层禁忌”（Decoding-Level Taboo），一种零提示诊断压力测试方法，它在运行时直接干预logit空间，迫使模型脱离标称路径。通过在词边界动态屏蔽主要候选词元，Taboo迫使机器进行迂回表达。在多个开放权重模型系列上对Taboo进行评估后发现，偏离路径的鲁棒性受参数规模和训练后指令对齐的显著影响，且鲁棒性通常随模型规模和指令对齐程度的提高而改善。除本文呈现的结果外，Taboo还为生成多样化合成数据集、对运行时安全护栏进行压力测试，以及在现实部署之前审计模型可靠性提供了一种新颖的原语。

English

Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimized generation corridor. In real-world deployments, however, complex system prompts, safety guardrails, and structural constraints continuously force models off this nominal path, driving a divergence between benchmark scores and deployment performance. To address this issue, we introduce Decoding-Level Taboo, a zero-prompt diagnostic stress test that intervenes directly in logit space at runtime, forcing models out of their nominal paths. By dynamically masking primary candidate tokens at word boundaries, Taboo forces machine circumlocution. Evaluating Taboo across several open-weight model families reveals that off-path robustness is heavily influenced by both parameter scale and post-training instruction alignment, with robustness generally improving with model size and alignment. Beyond the results presented in this paper, Taboo provides a novel primitive for generating diverse synthetic datasets, stress-testing runtime safety guardrails, and auditing model reliability prior to real-world deployment.