Agon: 竞争性跨模型强化学习与隐式对手推理评分

摘要

基于可验证奖励的强化学习（如GRPO）是当今推理模型的核心驱动力，但它仅对最终答案进行评分。在复杂问题上，这会训练模型写得更长而非想得更深，因为中间推理过程从未被评分，也不存在衡量优秀思维的标准。我们提出Agon方法，让两个相互竞争的模型互为评分者。两个模型尝试解决同一问题：在交替角色中，一个模型起草解答，另一个在阅读该解答的同时独立求解，每个模型因比对方更擅于解题而获得奖励。为了获胜，模型必须比看到其工作过程的对手推理得更优，因此推理能力在训练过程中被隐式评判，无需过程标签或奖励模型。由于两个模型同时优化，每个模型都面临不断变强的对手，这是单模型强化学习无法提供的。两者只需实力相当且行为模式各异即可。推理时，该配对按训练方式部署：形成两级级联，一个模型先起草，另一个阅读草案后再作答。在DeepMath的困难子集上结合Qwen3使用时，该方法使GRPO的pass@1翻倍，相较基于相同基座模型的未经训练的混合智能体方法，增益约达八倍。该结果在竞赛编程代码以及不同模型族（Qwen3.5、Gemma 4）上均得到复现。目前模型以文本形式交互；下一步将让它们在潜在空间中进行协同推理。

English

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.