Co-RL：多智能体强化学习中多样化群体涌现无监督推理

摘要

强化学习（RL）已成为提升语言模型及视觉-语言模型推理能力的强大方法，然而其最显著的成功仍高度依赖于真值监督（例如可验证奖励）。此类标注获取成本高昂，且随着推理能力超越人类可靠评估的范畴，其可获得性日益稀缺。自奖励强化学习通过使模型能够从自身生成结果中推导奖励信号，减少了对这类标注的依赖。然而，仅基于自生成反馈进行训练可能强化既有偏差与次优行为，降低响应多样性，并最终导致响应同质化和训练崩溃。在本工作中，我们证明无监督推理可通过协作式多智能体训练得以涌现。我们提出了Co-RL框架，其中多个不共享参数的解耦模型通过基于同伴生成的奖励进行强化学习，实现同步优化。我们进一步表明，通过异构模型系列、不同规模以及改写后的训练样本提升队列多样性，可减少驱动自我强化反馈循环的相关误差。这种多样性持续提升了推理性能，保持了行为多样性，并缓解了训练崩溃。在纯文本和多模态领域，Co-RL均持续优于基础模型及先前的无标签方法，同时达到或超越有监督方法，且无需访问任何真值标签。具体而言，Co-RL在针对LLM的七个纯文本基准上平均提升3.0%-8.6%，在针对VLM的四个多模态基准上平均提升2.3%-7.2%。代码可在https://github.com/DrStranded/Co-RL获取。

English

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.