Stability Issues in Small Language Model RL Training Identified

A recent investigation published on arXiv (2607.25091) delves into the alignment of reinforcement learning (RL) for Small Language Models (SLMs) with parameters ranging from 70 to 500 million, uncovering three distinct failure modes that can be reproduced. Researchers employed Proximal Policy Optimization (PPO) to train fifteen configurations of models and corpora, which included Pythia-70M, 160M, 410M, and SmolLM2-135M, 360M, using datasets like TinyStories, CNN/DailyMail, and Wikitext-103. The failures identified consist of silent LoRA parameter freezing within standard PEFT/TRL frameworks, numerical overflow in bfloat16 importance ratios, and catastrophic policy collapse due to reward-model inaccuracies. Proposed remedies involve a merge-and-reinitialize adapter method, utilizing float32 precision during PPO updates, and implementing a three-layer safety system with reward whitening and importance-ratio protection.