Repeating smaller datasets speeds up AI training via sampling biases

A recent study in machine learning indicates that utilizing smaller datasets with increased repetitions may be quicker and more efficient in terms of computation compared to larger datasets. This effect, known as the "small-vs-large gap," was noted across different algorithmic tasks, architectures, and optimizers. Researchers propose that the observed acceleration results from suitable layer-wise growth facilitated by sampling biases, which are more significant with smaller datasets. The findings offer both theoretical insights and empirical support, demonstrating that repeating smaller datasets can serve as a beneficial inductive bias for optimization, especially in reasoning tasks, rather than just a solution when data is limited. The research is accessible on arXiv.