壓縮短文本生成中的品質斷裂點：階段性瓶頸定位

摘要

壓縮短文本生成器可能在兩個不同環節失效：編解碼器可能在生成開始前就丟棄資訊，或者潛在生成器可能產生較弱的編碼。若未區分這些失效模式，研究人員可能將計算資源浪費在改進錯誤的組件上。我們在一個由層級式 VQ-VAE-2 編解碼器與遮罩離散擴散生成器（MDLM）構建的控制型 64 對 16 TinyStories 案例研究中探討此問題。我們採用分階段驗證協議，在共享的外部 GPT-2 評分器下分離編解碼器重建保真度、潛在生成品質及輔助潛在診斷指標，同時為幾何研究報告互補的語義指標。在測試配置中，僅編解碼器重建即導致外部困惑度中位數從 15.17 升至 27.36（+80.4%），第 95 百分位數從 25.10 升至 98.91（+294.1%），顯示主要品質損失發生在潛在生成開始之前。在同一評分器下，編碼空間 MDLM 仍顯著優於詞元空間擴散，平均值、中位數及第 95 百分位數分別降低 32.9%、30.9% 及 36.6%。幾何感知正規化改善了局部潛在代理指標，但在現有運行中未提升解碼後文本指標。本文貢獻在於方法論而非演算法：為具體管線提出可重複的分階段診斷方法，並證明在此設定下，編解碼器保真度而非潛在去噪過程設定了實際品質上限。

English

Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.