Co-RL：多智能體強化學習中多樣化群體湧現非監督式推理

摘要

強化學習（RL）已成為提升語言與視覺語言模型推理能力的強大方法，然而其最顯著的成功仍高度依賴於真實標籤監督（例如可驗證獎勵）。此類標註取得成本高昂，且隨著推理能力進展到人類難以可靠評估的程度，這類標註日益稀缺。自我獎勵強化學習透過讓模型從自身生成結果中推導獎勵訊號，降低了這種依賴。然而，僅依賴自我生成反饋進行訓練，可能強化既有偏見與次佳行為、降低回應多樣性，最終導致回應同質化與訓練崩潰。在本工作中，我們展示無監督推理可透過合作式多智能體訓練而湧現。我們提出 Co-RL 框架，其中多個不共享參數的解耦模型，利用來自同儕模型的獎勵，透過 RL 同時進行最佳化。我們進一步證明，透過異質模型家族、不同規模以及改寫後的訓練樣本提升群體多樣性，可減少驅動自我強化反饋迴圈的相關誤差。這種多樣性持續提升推理效能、維持行為多樣性，並減緩訓練崩潰。在純文字與多模態領域中，Co-RL 始終優於基礎模型與先前的無標籤方法，同時在無法取得任何真實標籤的情況下，達到或超越監督式方法的表現。具體而言，Co-RL 在 LLM 的七個純文字基準上平均提升 3.0-8.6%，在 VLM 的四個多模態基準上平均提升 2.3-7.2%。程式碼已公開於 https://github.com/DrStranded/Co-RL。

English

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.