压缩短文本生成中质量下降的位置：分阶段瓶颈定位

摘要

压缩短文本生成器可能在两个不同环节失败：编解码器可能在生成开始前丢弃信息，或者潜在生成器可能产生弱编码。若无法区分这两种失效模式，研究人员可能将计算资源浪费在改进错误组件上。我们通过一个基于分层VQ-VAE-2编解码器和掩码离散扩散生成器（MDLM）构建的64-to-16 TinyStories受控案例研究来探讨该问题。采用分阶段验证协议，在统一外部GPT-2评分器下分别评估编解码器重建保真度、潜在生成质量及辅助潜在诊断指标，同时为几何研究提供互补语义指标。在所测试配置中，仅编解码器重建就将外部困惑度中位数从15.17提升至27.36（+80.4%），95%分位数从25.10提升至98.91（+294.1%），表明主要质量损失出现在潜在生成开始之前。在相同评分器下，代码空间MDLM仍显著优于令牌空间扩散，使均值、中位数和95%分位数分别降低32.9%、30.9%和36.6%。几何感知正则化可改善局部潜在代理指标，但在现有运行中未提升解码文本指标。本文贡献在于方法论层面而非算法层面：针对具体管道提出可复用的分阶段诊断方法，并表明在该设置下，编解码器保真度而非潜在去噪过程决定了实际质量上限。

English

Compressed short-text generators can fail in two different places: the codec may discard information before generation starts, or the latent generator may produce weak codes. Without separating these failure modes, researchers can spend compute improving the wrong component. We study this problem in a controlled 64-to-16 TinyStories case study built from a hierarchical VQ-VAE-2 codec and a masked discrete diffusion generator (MDLM). We use a staged validation protocol that separates codec reconstruction fidelity, latent generation quality, and auxiliary latent diagnostics under one shared external GPT-2 scorer, while reporting complementary semantic metrics for the geometry study. In the tested configuration, codec reconstruction alone raises median external perplexity from 15.17 to 27.36 (+80.4%) and p95 from 25.10 to 98.91 (+294.1%), showing that the dominant quality loss appears before latent generation begins. Under the same scorer, code-space MDLM remains materially stronger than token-space diffusion, reducing mean, median, and p95 by 32.9%, 30.9%, and 36.6%, respectively. Geometry-aware regularization improves local latent proxies but does not improve decoded-text metrics in the available runs. The contribution is methodological rather than algorithmic: the paper presents a reusable staged diagnosis for one concrete pipeline and shows that, in this setting, codec fidelity rather than latent denoising sets the practical quality ceiling.