重新審視推測解碼中的有損驗證：機制、權衡與失敗模式

摘要

推測解碼（Speculative Decoding, SD）透過允許輕量級的草稿模型提出候選詞元，再由較大的目標模型進行平行驗證，以加速大型語言模型的推論。近期方法引入了有損驗證機制，透過放寬嚴格的分布匹配來進一步提升效率。然而，此類放寬會默默改寫解碼分布，所帶來的加速可能以不穩定、有時甚至嚴重劣化的生成品質為代價。在本研究中，我們對有損驗證方法所引發的分布提出了原則性分析。我們證明許多看似不同的方法僅在表面上有所差異，可歸類為兩大類：基於截斷的驗證與協作式驗證。我們進一步在精心挑選的基準測試上建構了一個診斷性評估框架。對於基於截斷的方法，我們發現一個根本性的陷阱：由於分布失真，其效能相較於真正的截斷取樣基準可能顯著下降。對於協作式驗證，我們揭示了一項關鍵原則：控制草稿機率相對目標機率的超出量，對於防止低品質輸出至關重要。我們的程式碼位於 https://github.com/ZhouYuxuanYX/Fast-HSD。

English

Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality. In this work, we present a principled analysis of the distributions induced by lossy verification methods. We show that many seemingly distinct approaches differ only superficially and can be classified into two categories: truncation-based verification and collaborative verification. We further construct a diagnostic evaluation framework across curated benchmarks. For truncation-based methods, we identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion. For collaborative verification, we uncover a key principles: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs. Our code is available at https://github.com/ZhouYuxuanYX/Fast-HSD.