重新审视投机解码中的有损验证：机制、权衡与失败模式

摘要

推测解码（Speculative Decoding, SD）通过允许轻量级草稿模型提出词元，并由更大的目标模型并行验证，从而加速大语言模型推理。近期方法引入有损验证方案，通过放宽严格的分布匹配来进一步提升效率。然而，这种放宽会悄然改写解码分布，由此带来的加速可能以不稳定的、有时甚至严重退化的生成质量为代价。在本工作中，我们对有损验证方法所诱导的分布进行了原则性分析。我们表明，许多看似不同的方法仅在表面上有所区别，可归类为两种类型：基于截断的验证和协作验证。我们进一步在精选基准上构建了诊断评估框架。对于基于截断的方法，我们发现了一个根本性陷阱：由于分布扭曲，其性能相较于真实的截断采样基线可能显著下降。对于协作验证，我们揭示了一个关键原则：控制草稿概率相对于目标概率的过冲（overshoot）对防止低质量输出至关重要。我们的代码可在 https://github.com/ZhouYuxuanYX/Fast-HSD 获取。

English

Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality. In this work, we present a principled analysis of the distributions induced by lossy verification methods. We show that many seemingly distinct approaches differ only superficially and can be classified into two categories: truncation-based verification and collaborative verification. We further construct a diagnostic evaluation framework across curated benchmarks. For truncation-based methods, we identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion. For collaborative verification, we uncover a key principles: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs. Our code is available at https://github.com/ZhouYuxuanYX/Fast-HSD.