Study Reveals Post-Training Reduces Language Model Output Diversity, Impacting Creative AI Applications
A recent study investigates the impact of post-training processes on the diversity of outputs in language models, focusing on three parallel branches of the Olmo 3 model. The research analyzes Think (chain-of-thought distillation), Instruct (extensive multi-source data), and RL-Zero methods across 15 tasks using four metrics for text diversity. Findings indicate that post-trained models yield less varied outputs compared to their original versions, which threatens inference-time scaling techniques reliant on diverse samples. This decline in output diversity could lead to uniformity in creative and value-sensitive tasks. The study reveals that the Think lineage experiences the most semantic diversity loss during supervised fine-tuning. The effects of Direct Preference Optimization (DPO) are more pronounced in Instruct models than in Think models. Additionally, limiting chain-of-thought reasoning in Think models further diminishes output diversity. This research challenges previous assumptions that linked output collapse solely to specific post-training techniques, distinguishing the influence of training data composition from the methods used and the generation format from model weights. The findings are detailed in arXiv:2604.16027v1 with an announcement type of cross.
Key facts
- Post-trained language models produce less varied outputs than base models
- Output diversity collapse undermines inference-time scaling methods
- Collapse risks homogenizing model outputs on creative and value-laden tasks
- Study analyzes three parallel post-training lineages of Olmo 3 model
- Research examines Think, Instruct, and RL-Zero approaches across 15 tasks
- Location of collapse co-varies with training data composition
- Think lineage loses most semantic diversity at supervised fine-tuning
- Effect of DPO is larger in Instruct than in Think models
Entities
—