RL Post-Training Compression Reduces LLM Overthinking

A recent study published on arXiv (2605.07316) indicates that reinforcement learning utilizing verifiable rewards enhances the reasoning capabilities of large language models (LLMs), but frequently leads to excessive deliberation—resulting in unnecessarily lengthy reasoning processes. Current solutions, such as length penalties or early-exit methods, may compromise accuracy or lead to insufficient reasoning. Through an examination of training dynamics, the researchers discovered that the correlation between length and accuracy initially shows a negative trend (overthinking) before shifting to a positive one (underthinking) during compression. To address this, they suggest employing implicit compression regularization to facilitate succinct reasoning without these drawbacks.