Self-Training Restructures Language, Not Just Flattens It

A recent study disputes the common belief that repeated self-training on a language model's outputs leads to a decrease in linguistic diversity and a flattening of language. Researchers examined eleven generations of self-training across five models (GPT-2 124M, Pythia-410M, Pythia-1.4B, OPT-1.3B, Pythia-2.8B) and discovered that language undergoes restructuring rather than mere flattening. They observed an increase in surface markers like discourse connectives, hedges, and em-dashes, while more complex syntactic structures, such as questions, parentheticals, passives, and subjunctives, diminish. This phenomenon is termed the Structural Depth Hypothesis (SDH), suggesting that the decay rate of a linguistic feature correlates primarily with its structural depth. The research utilizes 17-feature panels from the five models and is accessible on arXiv with reference 2605.20602.