COSE: Confidence-Weighted PPO for Self-Evolving LLMs

A novel method called COSE (Confidence-Orchestrated Self-Evolution) has been introduced by researchers to enable self-evolving large language models (LLMs). This technique utilizes the model's inherent confidence as a subtle uncertainty indicator to enhance learning. COSE incorporates confidence-weighted PPO updates and confidence-prioritized replay, tackling the issue of training signals where incorrect self-assessments result in flawed gradient updates. It eliminates the need for external validators or considering noisy self-generated feedback as supervision. When tested on 19 held-out benchmarks and four Qwen/Llama backbones (ranging from 0.6B to 4B), COSE consistently outperforms base models, achieving the highest average performance in general reasoning and mathematics. The findings are available on arXiv under ID 2605.28010.