Agon：具備隱式對手推理評分的競爭性跨模型強化學習

摘要

基於可验证奖励（如GRPO）的強化學習是當今推理模型的驅動力，然而它僅對最終答案進行評分。在處理難題時，這種方法訓練模型寫得更多而非思考得更好，因為推理過程本身從未被評分，且沒有標註何謂良好的思考。我們提出Agon，讓兩個相互競爭的模型互為對方的評分者。兩者嘗試解決同一問題；在交替角色中，一個模型起草解答，另一個則在解題時閱讀該草案，每個模型因勝過對方而獲得獎勵。為了取勝，模型必須在推理上勝過一個已看過其工作成果的對手，因此推理能力在訓練過程中隱式地被評判，無需過程標籤，也無需獎勵模型。由於兩個模型都被優化，每個模型都面臨日益強大的對手，這是單一模型強化學習所無法提供的。兩者只需實力相當且行為上存在差異。在推理階段，這對模型以訓練時的方式部署——一種兩階段級聯模式：一個模型起草，另一個在閱讀草案後給出答案。在Deepmath的困難子集上（使用Qwen3），該方法使GRPO的pass@1翻倍，約為未經訓練的混合智能體相對於同一基礎模型所獲增益的八倍。這一結果在競賽編程代碼及不同模型家族（Qwen3.5、Gemma 4）中均得到復現。目前，模型以文本形式交流；下一步是讓它們在潛在空間中共同推理。

English

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.