Grokking Dynamics Explained via Weight-Decay Clock and Broken Symmetries
A recent theoretical study published on arXiv (2607.23967) presents a model that can be precisely solved for grokking, which refers to the delayed generalization phenomenon observed in neural networks. The researchers pinpoint a "grokking subspace" within the empirical null space, where predictions during training do not change, making weight decay the only force that restores balance. This results in a gradual dissipative relaxation, dictated by both discrete-time and continuous-time laws. They provide specific iteration-scale predictions for grokking time, recovering the known (1-β)/(ηλ) scaling in the weak-regularization context. The findings are relevant to linear models utilizing full-batch heavy-ball optimization and weight decay, with a locally quadratic extension applicable to nonlinear networks, offering insights into grokking through softly broken symmetries.
Key facts
- Paper on arXiv: 2607.23967
- Grokking is delayed generalization in neural networks
- Identifies a grokking subspace in the empirical null space
- Weight decay is the sole restoring force in that subspace
- Derives exact discrete-time and continuous-time relaxation laws
- Grokking time scales as (1-β)/(ηλ) in weak-regularization regime
- Analysis uses full-batch heavy-ball optimization
- Extension to nonlinear networks via locally quadratic approximation
Entities
Institutions
- arXiv