X-AuT：基于跨尺度蒸馏的语音大语言模型渐进式音频编码器压缩

摘要

降低音频编码器深度可降低语音大语言模型的推理成本，但移除完整模块会扰动解码器所消费的嵌入表示，并可能导致删除错误与过早序列结束错误。我们提出 X-AuT，一个渐进式框架，通过简短行为探测选择层组合，并通过表征对齐、跨尺度蒸馏、调度式学生策略监督和 LoRA 微调来恢复剪枝后的模型。语言模型主干保持冻结，而注意力 LoRA 适配器与绑定输出嵌入在蒸馏过程中进行适配。训练使用来自转写一致性流水线的最高一致性档位，随后在微调期间进行数据源重加权。在十个公开中英基准上，将 Qwen3-ASR-0.6B 从 18 层音频编码器压缩至 16 层，可使宏平均错误率从 5.61% 降至 5.27%。14 层模型达到 5.75%，同时音频塔参数减少 20.7%。在匹配配方下，1.7B 教师模型取得 5.55% 的平均错误率，而自蒸馏为 8.45%；渐进式 18→14 剪枝优于直接剪枝（5.75% vs. 6.73%）。这些单次运行结果确立了两种实用工作点，并表明准确率影响在不同基准间存在差异。项目网站：https://xpeng-ai.github.io/x-aut

English

Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18rightarrow14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut