以演化式評分標準作為獎勵的強化學習於音訊推理

摘要

音訊推理對於機器理解聲學世界至關重要。具有可驗證獎勵的強化學習能夠誘發此類推理，然而現有獎勵設計的局限性是互補的：基於結果的獎勵僅監督最終答案，使模型在未關注音訊的情況下即可得出答案；而基於過程的獎勵則對推理過程本身評分，但依賴於粗略、人工設計且固定的標準，既無法適應每個問題，也無法始終立足於聲學證據。此外，問題的要求各不相同，有些取決於感知，有些取決於多步推理，而任何靜態標準都會隨著策略的改進而失效。因此，以細粒度、以音訊為依據且自適應的獎勵來監督推理過程至關重要，但這也具有挑戰性，因為針對每個樣本人工設計此類獎勵並不切實際。為此，我們提出了 AudioRubrics，這是一個以自我演化、以音訊為依據的評分標準獎勵來監督音訊推理的強化學習框架。AudioRubrics 從原始波形合成每個樣本的評分標準，並以模型自身的生成軌跡為條件，按群組重新生成並重新加權標準，提供持續的學習信號，從而在靜態標準飽和之際持續針對當前策略的弱點。在三個音訊推理基準上的全面評估顯示，AudioRubrics 大幅優於眾多開源及基於訓練的基線。此外，我們的分析表明，收益隨評分標準生成器與評判器的能力而擴大，且 AudioRubrics 收斂到穩定的推理長度，既避免退化性崩潰，也避免無界增長。音訊感知的改善進一步證明了將監督錨定於聲學證據的有效性。我們的專案頁面位於 https://audiorubrics.github.io。

English

Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.