Scaling Laws in Shallow Neural Networks Linked to Spectral Properties
A recent theoretical investigation released on arXiv (2509.24882) thoroughly examines scaling laws applicable to quadratic and diagonal neural networks within the feature learning context. By utilizing relationships with matrix compressed sensing and LASSO, the researchers create an elaborate phase diagram that illustrates scaling exponents of excess risk in relation to sample complexity and weight decay. Their findings reveal transitions between various scaling regimes and plateau behaviors, reflecting trends commonly noted in empirical studies of neural scaling. Importantly, this research establishes a clear connection between these regimes and the spectral characteristics of trained network weights, thereby providing theoretical support for recent empirical findings that link power-law spectral behavior to scaling laws. This study broadens the theoretical framework beyond linear models, aiding in the interpretation of neural scaling phenomena.
Key facts
- The study focuses on quadratic and diagonal neural networks in the feature learning regime.
- It uses connections with matrix compressed sensing and LASSO.
- A phase diagram for scaling exponents of excess risk is derived.
- Crossovers between distinct scaling regimes and plateau behaviors are identified.
- A link between scaling regimes and spectral properties of trained weights is established.
- The work provides theoretical validation of empirical observations on power-law spectra.
- The paper is available on arXiv with ID 2509.24882.
Entities
Institutions
- arXiv