CADENCE：通过覆盖自适应在策略蒸馏缩小推理差距

摘要

同策略知识蒸馏将推理能力从大型教师模型迁移至紧凑学生模型，但现有方法存在三种复合失效模式：（i）冷启动崩溃——新初始化的学生模型对教师模型偏好的token赋予近乎零的概率；（ii）状态无关的散度调度——仅依赖时间步的前向/反向KL插值忽略学生模型的覆盖状态；（iii）二元奖励稀疏性——仅通过通过/失败信号丢弃部分正确轨迹中的信息。本文提出CADENCE统一框架，针对每种失效模式提出精准修复方案。其DRIFT机制在学生采样轨迹上，对每个token构建前向KL与反向KL替代目标的凸组合（采用逐token替代目标，而非序列级KL梯度估计器）。框架包含六个扩展组件：（A）COVA——覆盖自适应β调度机制，加速前向到反向的转换；（B）FTB——分叉token增强，通过全局归一化熵参考在高熵位置集中梯度；（C）CCD——密集奖励，为错误但接近的轨迹添加数值邻近性部分正确分数；（D）LAP——简洁优先的正确轨迹强化；（E）EMR——熵匹配校准正则化器；（F）BSD——自举自蒸馏阶段。在GSM8K和MATH-500数据集上（采用修正的512token协议，5个随机种子的标准差报告），CADENCE将0.5B参数学生模型从1.5B教师模型中蒸馏，在GSM8K pass@1指标上达到69.8%±0.5%（预训练模型为48.7%，弥补教师差距的63.2%）；使用3B教师模型时达到72.1%±0.4%（弥补差距76.2%），以超过最强匹配计算量的标签基线（DRIFT+二元奖励）4.4%±0.7个百分点。所有实验在单一Apple Mac Studio（M系列芯片，64GB统一内存）上完成，表明基于原则的知识蒸馏无需数据中心级硬件即可实现强推理质量。

English

On-policy knowledge distillation transfers reasoning from large teachers to compact students, but existing approaches suffer three compounding failure modes: (i) cold-start collapse, where a fresh student assigns near-zero mass to teacher-preferred tokens; (ii) state-agnostic divergence scheduling, where time-only forward/reverse-KL interpolation ignores the student's coverage state; and (iii) binary reward sparsity, where pass/fail signals discard information from partially correct traces. We present CADENCE, a unified framework with a targeted fix for each. Its DRIFT mechanism schedules a per-token convex mixture of forward-KL and reverse-KL surrogate objectives on student-sampled trajectories (per-token surrogates, not sequence-level KL gradient estimators). Six components extend it: (A) COVA, a coverage-adaptive β schedule accelerating the forward-to-reverse transition; (B) FTB, a forking-token boost concentrating gradient at high-entropy positions via a globally-normalized entropy reference; (C) CCD, a dense reward adding numerical-proximity partial credit for incorrect-but-close traces; (D) LAP, brevity-preferential correct-rollout reinforcement; (E) EMR, an entropy-matching calibration regularizer; (F) BSD, a bootstrapped self-distillation phase. On GSM8K and MATH-500 (corrected 512-token protocol, 5 seeds, reported std), CADENCE distills a 0.5B student from a 1.5B teacher to 69.8 pm 0.5% GSM8K pass@1 (from 48.7% pretrained; 63.2% of the teacher gap closed) and to 72.1 pm 0.4% with a 3B teacher (76.2% closed), beating the strongest matched-compute label-using baseline (DRIFT+binary reward) by +4.4 pm 0.7 points. All experiments run on a single Apple Mac Studio (M-series, 64GB unified memory), showing principled distillation reaches strong reasoning quality without datacenter-scale hardware.