Optimal Data Repetition in LLM Pretraining Scales with Model Size

A recent paper on arXiv (2608.14071) explores the optimal strategy for repeating high-quality domain data during the pretraining phase of large language models (LLMs) as both model sizes and training-token budgets expand. The researchers discovered that, unexpectedly, the ideal repetition count for a specific domain slightly increases with model size when maintaining a constant tokens-per-parameter ratio (TPP). Furthermore, across various domains, a strong negative correlation exists between the optimal repetition count and final validation loss; domains with lower loss typically benefit from increased repetition. This study tackles the issue of limited high-quality domain data, which diminishes in proportion as budgets rise, suggesting repetition as a solution while warning against the risk of overfitting. The paper has been announced as a new submission on arXiv.