Co-RL: Multi-Agent RL Framework for Unsupervised Reasoning in AI Models
Co-RL, a framework from arXiv preprint 2608.17253, proposes that unsupervised reasoning can emerge through cooperative multi-agent training. Reinforcement learning (RL) often improves reasoning in language and vision-language models but typically requires costly ground-truth supervision, such as verifiable rewards, which become scarce as capabilities advance. Self-rewarding RL reduces this dependency by using a model's own completions, yet it risks reinforcing biases, reducing diversity, and causing training collapse. Co-RL instead optimizes multiple decoupled models—sharing no parameters—simultaneously through RL, with rewards derived from the cohort. This approach aims to bypass the limitations of self-rewarding RL.
Key facts
- Co-RL is introduced as a multi-agent reinforcement learning framework.
- Multiple decoupled models share no parameters.
- Models are optimized simultaneously through RL.
- The goal is to enable unsupervised reasoning.
- RL is used to improve reasoning in language and vision-language models.
- Strong RL successes depend on ground-truth supervision like verifiable reward.
- Self-rewarding RL can reinforce biases, reduce diversity, and lead to training collapse.
- Cooperative multi-agent training is proposed as a solution.
Entities
—