Flat Minima in Neural Networks Are an Illusion, Study Shows

A recent study published on arXiv disputes the common assumption that flat minima in the loss landscapes of neural networks enhance generalization. The author reveals that reparameterization preserving function can significantly increase the Hessian of any minimum without changing predictions, suggesting that flatness is not a causal element. Instead, the research introduces the concept of "weakness," defined as the volume of completions that align with the learned function in the learner's embodied language, as the actual factor influencing generalization. Weakness remains invariant under reparameterization and is shown to be minimax-optimal for exchangeable demands. Additionally, the findings indicate that PAC-Bayes bounds are effective due to their correlation with weakness, with experiments on MNIST demonstrating the advantages of large-batch generalization.