以演化评分标准为奖励的音频推理强化学习

摘要

音频推理对于机器理解声学世界至关重要。基于可验证奖励的强化学习能够激发此类推理，然而现有奖励设计在局限性上具有互补性：基于结果的奖励仅监督最终答案，使模型无需关注音频即可得出答案；而基于过程的奖励虽然对推理过程本身进行评分，却依赖于粗糙、手工设计且固定的标准，既无法适应每个问题，也难以锚定于声学证据。此外，不同问题对能力的要求各不相同，有的依赖于感知，有的依赖于多步推理，而任何静态标准都会随着策略的改进而逐渐失效。因此，使用细粒度、音频锚定且自适应的奖励来监督推理过程至关重要，但也极具挑战性，因为这类奖励难以针对每个样本手工设计。为此，我们提出AudioRubrics——一种使用自演化、音频锚定的评分标准奖励来监督音频推理的强化学习框架。AudioRubrics从原始波形中合成为每个样本定制的评分标准，并以模型自身的采样轨迹为条件，按组重新生成并重新加权这些标准，从而提供持续的学习信号，使静态标准趋于饱和时仍能持续针对当前策略的弱点进行优化。对三个音频推理基准的全面评估表明，AudioRubrics显著优于多种开源及基于训练的基线方法。此外，我们的分析表明，性能提升随评分标准生成器和评判器能力的增强而扩展，并且AudioRubrics会收敛到稳定的推理长度，既避免了退化性坍缩，也避免了无界增长。音频感知能力的提升进一步证明了将监督锚定于声学证据的有效性。我们的项目页面可在 https://audiorubrics.github.io 获取。

English

Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.