CADENCE：透過覆蓋率自適應的在線策略蒸餾填補推理差距

摘要

在线策略知识蒸馏将推理能力从大模型迁移至紧凑学生模型，但现有方法存在三种复合失效模式：（i）冷启动崩溃，即新初始化的学生模型对教师模型偏好的令牌赋予近乎零的概率；（ii）状态无关的发散调度，即仅依赖时间的前向/反向KL插值忽略了学生模型的覆盖状态；（iii）二元奖励稀疏性，即通过/失败信号丢弃了部分正确轨迹中的信息。我们提出CADENCE，一个针对每种失效模式提供针对性修复的统一框架。其DRIFT机制在学生模型采样的轨迹上（基于逐令牌替代目标，而非序列级KL梯度估计器）调度前向KL与反向KL替代目标的逐令牌凸组合。六大扩展组件进一步强化该机制：（A）COVA——覆盖度自适应β调度，加速从正向到反向的过渡；（B）FTB——分叉令牌增强，通过全局归一化熵参考将梯度集中于高熵位置；（C）CCD——密集奖励，为错误但接近的轨迹添加数值邻近部分正确得分；（D）LAP——偏好简洁的正确轨迹强化；（E）EMR——熵匹配校准正则化器；（F）BSD——自举式自我蒸馏阶段。在GSM8K和MATH-500数据集上（采用修正的512令牌协议，5个随机种子，报告标准差），CADENCE从1.5B教师模型蒸馏出0.5B学生模型，在GSM8K上达到69.8±0.5%的pass@1（预训练模型为48.7%；教师-学生差距的63.2%被弥合）；当使用3B教师模型时达到72.1±0.4%（差距弥合76.2%），以+4.4±0.7个百分点的优势超越最强的匹配计算标签使用基线（DRIFT+二元奖励）。所有实验均在单台Apple Mac Studio（M系列，64GB统一内存）上运行，表明理论驱动的蒸馏方法无需数据中心级硬件即可达到强大的推理质量。

English

On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We present CADENCE, a unified framework with a targeted fix for each. Its DRIFT mechanism schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories (per-token surrogates, not sequence-level KL gradient estimators). Six components extend it: (A) COVA, a coverage-adaptive β schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (C) CCD, a dense reward adding numerical-proximity partial credit for incorrect-but-close traces; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-matching calibration regularizer; (F) BSD, a bootstrapped self-distillation phase. On GSM8K and MATH-500 (corrected 512-token protocol, 5 seeds, reported std), CADENCE distills a 0.5B student from a 1.5B teacher to 69.8 pm 0.5% GSM8K pass@1 (from 48.7% pretrained; 63.2% of the teacher gap closed) and to 72.1 pm 0.4% with a 3B teacher (76.2% closed), beating the strongest matched-compute label-using baseline (DRIFT+binary reward) by +4.4 pm 0.7 points. All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.